[FIXED] Write multi-line secret to windows runner in GitHub workflow

Issue

Summary

What specific syntax must be changed in the code below in order for the multi-line contents of the $MY_SECRETS environment variable to be 1.) successfully written into the C:\\Users\\runneradmin\\somedir\\mykeys.yaml file on a Windows runner in the GitHub workflow whose code is given below, and 2.) read by the simple Python 3 main.py program given below?

PROBLEM DEFINITION:

The echo "$MY_SECRETS" > C:\\Users\\runneradmin\\somedir\\mykeys.yaml command is only printing the string literal MY_SECRETS into the C:\\Users\\runneradmin\\somedir\\mykeys.yaml file instead of printing the multi-line contents of the MY_SECRETS variable.

We confirmed that this same echo command does successfully print the same multi-line secret in an ubuntu-latest runner, and we manually validated the correct contents of the secrets.LIST_OF_SECRETS environment variable. ... This problem seems entirely isolated to either the windows command syntax, or perhaps to the windows configuration of the GitHub windows-latest runner, either of which should be fixable by changing the workflow code below.

EXPECTED RESULT:

The multi-line secret should be printed into the C:\\Users\\runneradmin\\somedir\\mykeys.yaml file and read by main.py.

The resulting printout of the contents of the C:\\Users\\runneradmin\\somedir\\mykeys.yaml file should look like:

***  
***  
***  
***  

LOGS THAT DEMONSTRATE THE FAILURE:

The result of running main.py in the GitHub Actions log is:

ccc item is:  $MY_SECRETS

As you can see, the string literal $MY_SECRETS is being wrongly printed out instead of the 4 *** secret lines.

REPO FILE STRUCTURE:

Reproducing this error requires only 2 files in a repo file structure as follows:

.github/
    workflows/
        test.yml
main.py   

WORKFLOW CODE:

The minimal code for the workflow to reproduce this problem is as follows:

name: write-secrets-to-file
on:
  push:
    branches:
      - dev
jobs:
  write-the-secrets-windows:
    runs-on: windows-latest
    steps:
      - uses: actions/checkout@v3
      - shell: python
        name: Configure agent
        env:
          MY_SECRETS: ${{ secrets.LIST_OF_SECRETS }}
        run: |
          import subprocess
          import pathlib
          pathlib.Path("C:\\Users\\runneradmin\\somedir\\").mkdir(parents=True, exist_ok=True)
          print('About to: echo "$MY_SECRETS" > C:\\Users\\runneradmin\\somedir\\mykeys.yaml')
          output = subprocess.getoutput('echo "$MY_SECRETS" > C:\\Users\\runneradmin\\somedir\\mykeys.yaml')
          print(output)
          os.chdir('D:\\a\\myRepoName\\')
          mycmd = "python myRepoName\\main.py"
          p = subprocess.Popen(mycmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
          while(True):
            # returns None while subprocess is running
            retcode = p.poll() 
            line = p.stdout.readline()
            print(line)
            if retcode is not None:
              break 

MINIMAL APP CODE:

Then the minimal main.py program that demonstrates what was actually written into the C:\\Users\\runneradmin\\somedir\\mykeys.yaml file is:

with open('C:\\Users\\runneradmin\\somedir\\mykeys.yaml') as file:
  for item in file:
    print('ccc item is: ', str(item))
    if "var1" in item:
      print("Found var1")

STRUCTURE OF MULTI-LINE SECRET:

The structure of the multi-line secret contained in the secrets.LIST_OF_SECRETS environment variable is:

var1:value1
var2:value2
var3:value3
var4:value4

These 4 lines should be what gets printed out when main.py is run by the workflow, though the print for each line should look like *** because each line is a secret.

Solution

Edit: updated with fixed main.py and how to run it.

You can write the key file directly with Python:

      - shell: python
        name: Configure agent
        env:
          MY_SECRETS: ${{ secrets.LIST_OF_SECRETS }}
        run: |
          import os
          import pathlib
          pathlib.Path('C:\\Users\\runneradmin\\somedir\\').mkdir(parents=True, exist_ok=True)
          with open('C:\\Users\\runneradmin\\somedir\\mykeys.yaml', 'w') as key_file:
            key_file.write(os.environ['MY_SECRETS'])
      - uses: actions/checkout@v3
      - name: Run main
        run: python main.py

To avoid newline characters in your output, you need a main.py that removes the newlines (here with .strip().splitlines()):

main.py

with open('C:\\Users\\runneradmin\\somedir\\mykeys.yaml') as file:
    for item in file.read().strip().splitlines():
        print('ccc item is: ', str(item))
        if "var1" in item:
            print("Found var1")

Here's the input:

LIST_OF_SECRETS = '
key:value
key2:value
key3:value
'

And the output:

ccc item is:  ***
Found var1
ccc item is:  ***
ccc item is:  ***
ccc item is:  ***

Here is my complete workflow:

name: write-secrets-to-file
on:
  push:
    branches:
      - master
jobs:
  write-the-secrets-windows:
    runs-on: windows-latest
    steps:
      - shell: python
        name: Configure agent
        env:
          MY_SECRETS: ${{ secrets.LIST_OF_SECRETS }}
        run: |
          import os
          import pathlib
          pathlib.Path('C:\\Users\\runneradmin\\somedir\\').mkdir(parents=True, exist_ok=True)
          with open('C:\\Users\\runneradmin\\somedir\\mykeys.yaml', 'w') as key_file:
            key_file.write(os.environ['MY_SECRETS'])
      - uses: actions/checkout@v3
      - name: Run main
        run: python main.py

Also, a simpler version using only Windows shell (Powershell):

      - name: Create key file
        env:
          MY_SECRETS: ${{ secrets.LIST_OF_SECRETS }}
        run: |
          mkdir C:\\Users\\runneradmin\\somedir
          echo "$env:MY_SECRETS" > C:\\Users\\runneradmin\\somedir\\mykeys.yaml
      - uses: actions/checkout@v3
      - name: Run main
        run: python main.py

Answered By - duthils