How To Change Value In Array Bash

Although software engineers regularly use the command line for many aspects of development, arrays are likely i of the more obscure features of the command line (although non as obscure as the regex operator =~). But obscurity and questionable syntax aside, Bash arrays tin exist very powerful.

Expect, merely why?

Writing about Bash is challenging because it'southward remarkably easy for an article to devolve into a manual that focuses on syntax oddities. Balance assured, withal, the intent of this article is to avert having you RTFM.

A real (really useful) example

To that end, let's consider a real-world scenario and how Bash tin help: You lot are leading a new try at your company to evaluate and optimize the runtime of your internal data pipeline. As a start step, you want to do a parameter sweep to evaluate how well the pipeline makes use of threads. For the sake of simplicity, we'll treat the pipeline every bit a compiled C++ blackness box where the but parameter nosotros tin can tweak is the number of threads reserved for data processing: ./pipeline --threads 4.

The nuts

The start affair we'll do is ascertain an array containing the values of the --threads parameter that we desire to test:

                                    allThreads=(              one              2              4              viii              xvi              32              64              128              )

In this case, all the elements are numbers, but it need non be the case—arrays in Bash can contain both numbers and strings, east.chiliad.,myArray=(1 2 "three" four "five") is a valid expression. And just as with any other Bash variable, make sure to leave no spaces around the equal sign. Otherwise, Bash will treat the variable name as a program to execute, and the = every bit its first parameter!

At present that we've initialized the array, let'south retrieve a few of its elements. Yous'll detect that only doing echo $allThreads volition output only the first element.

To sympathise why that is, let's take a step back and revisit how nosotros unremarkably output variables in Bash. Consider the following scenario:

                                    type="commodity"              
              echo              "Found 42                $type"

Say the variable $type is given to the states as a singular noun and we want to add an s at the terminate of our sentence. We tin can't simply add together an s to $blazon since that would turn it into a unlike variable,$types. And although nosotros could apply lawmaking contortions such as echo "Found 42 "$type"s", the best way to solve this problem is to use curly braces: repeat "Found 42 ${blazon}s", which allows the states to tell Fustigate where the name of a variable starts and ends (interestingly, this is the same syntax used in JavaScript/ES6 to inject variables and expressions in template literals).

So as it turns out, although Bash variables don't mostly require curly brackets, they are required for arrays. In turn, this allows us to specify the index to access, e.1000.,echo ${allThreads[one]} returns the second chemical element of the array. Not including brackets, e.g.,echo $allThreads[1], leads Fustigate to care for [1] as a cord and output it as such.

Yes, Bash arrays accept odd syntax, simply at least they are aught-indexed, dissimilar another languages (I'm looking at you, R).

Looping through arrays

Although in the examples above nosotros used integer indices in our arrays, let's consider two occasions when that won't be the case: First, if nosotros wanted the $i-th element of the array, where $i is a variable containing the alphabetize of interest, we can retrieve that element using: echo ${allThreads[$i]}. 2nd, to output all the elements of an array, we supplant the numeric index with the @ symbol (you tin think of @ as standing for all): echo ${allThreads[@]}.

Looping through array elements

With that in heed, let'south loop through $allThreads and launch the pipeline for each value of --threads:

                                    for              t              in              ${allThreads[@]};              do              
              ./pipeline              --threads              $t              
              done

Looping through array indices

Next, permit'due south consider a slightly unlike approach. Rather than looping over array elements, nosotros tin can loop over assortment indices:

                                    for              i              in              ${!allThreads[@]};              do              
              ./pipeline              --threads              ${allThreads[$i]}              
              washed

Let'south break that down: Every bit we saw in a higher place, ${allThreads[@]} represents all the elements in our array. Adding an exclamation marking to make it ${!allThreads[@]} will return the list of all array indices (in our example 0 to 7). In other words, the for loop is looping through all indices $i and reading the $i-th element from $allThreads to set the value of the --threads parameter.

This is much harsher on the eyes, and so you may exist wondering why I bother introducing it in the showtime place. That's because there are times where you need to know both the alphabetize and the value within a loop, eastward.g., if you want to ignore the first element of an assortment, using indices saves you from creating an additional variable that you so increment inside the loop.

Populating arrays

So far, nosotros've been able to launch the pipeline for each --threads of interest. Now, let's assume the output to our pipeline is the runtime in seconds. We would like to capture that output at each iteration and save it in another array so we tin do various manipulations with it at the end.

Some useful syntax

But before diving into the code, we need to introduce some more syntax. First, nosotros need to be able to recollect the output of a Bash control. To exercise and then, use the post-obit syntax: output=$( ./my_script.sh ), which will store the output of our commands into the variable $output.

The second flake of syntax nosotros need is how to suspend the value we only retrieved to an array. The syntax to do that will expect familiar:

                      myArray+=(              "newElement1"              "newElement2"              )

The parameter sweep

Putting everything together, here is our script for launching our parameter sweep:

                                    allThreads=(              1              2              4              eight              16              32              64              128              )              
              allRuntimes=(              )              
              for              t              in              ${allThreads[@]};              exercise              
              runtime=$(./pipeline              --threads              $t              )              
              allRuntimes+=(              $runtime              )              
              washed

And voilà!

What else y'all got?

In this article, we covered the scenario of using arrays for parameter sweeps. But I promise there are more reasons to use Bash arrays—here are two more than examples.

Log alerting

In this scenario, your app is divided into modules, each with its own log file. We tin can write a cron job script to e-mail the right person when there are signs of trouble in sure modules:

                                    # List of logs and who should be notified of issues              
              logPaths=(              "api.log"              "auth.log"              "jenkins.log"              "data.log"              )              
              logEmails=(              "jay@electronic mail"              "emma@e-mail"              "jon@email"              "sophia@email"              )            
                          # Look for signs of trouble in each log              
              for              i              in              ${!logPaths[@]};
              practise              
              log=${logPaths[$i]}              
              stakeholder=${logEmails[$i]}              
              numErrors=$(              tail              -n              100              "$log"              |              grep              "Mistake"              |              wc              -fifty              )            
                          # Warn stakeholders if recently saw > 5 errors              
              if              [              [              "$numErrors"              -gt              5              ]              ];
              then              
              emailRecipient="$stakeholder"              
              emailSubject="Warning:                ${log}                showing unusual levels of errors"              
              emailBody="${numErrors}                errors establish in log                ${log}"              
              repeat              "$emailBody"              |              mailx              -southward              "$emailSubject"              "$emailRecipient"              
              fi              
              washed

API queries

Say you desire to generate some analytics about which users comment the most on your Medium posts. Since we don't have direct database access, SQL is out of the question, but we can use APIs!

To avert getting into a long discussion about API authentication and tokens, we'll instead use JSONPlaceholder, a public-facing API testing service, as our endpoint. Once we query each post and recall the emails of everyone who commented, we can append those emails to our results assortment:

                                    endpoint="https://jsonplaceholder.typicode.com/comments"              
              allEmails=(              )            
                          # Query first 10 posts              
              for              postId              in              {              one..x              };
              practice              
              # Make API phone call to fetch emails of this posts'due south commenters              
              response=$(curlicue              "${endpoint}?postId=${postId}"              )            
                          # Use jq to parse the JSON response into an array              
              allEmails+=(              $(              jq              '.[].e-mail'              <<<              "$response"              )              )              
              done

Note here that I'k using the jq tool to parse JSON from the command line. The syntax of jq is beyond the scope of this article, but I highly recommend yous expect into it.

As you might imagine, there are endless other scenarios in which using Bash arrays can help, and I hope the examples outlined in this article have given you some food for thought. If you lot have other examples to share from your own piece of work, delight leave a comment below.

But wait, there's more!

Since nosotros covered quite a bit of assortment syntax in this article, here's a summary of what we covered, forth with some more than advanced tricks we did not cover:

Syntax	Result
`arr=()`	Create an empty array
`arr=(1 two three)`	Initialize array
`${arr[ii]}`	Retrieve third element
`${arr[@]}`	Call back all elements
`${!arr[@]}`	Recollect array indices
`${#arr[@]}`	Calculate array size
`arr[0]=3`	Overwrite 1st element
`arr+=(4)`	Append value(south)
`str=$(ls)`	Save `ls` output as a string
`arr=( $(ls) )`	Save `ls` output as an assortment of files
`${arr[@]:south:n}`	Retrieve due north elements `starting at alphabetize southward`

Ane terminal thought

As we've discovered, Bash arrays sure accept foreign syntax, but I hope this article convinced you that they are extremely powerful. Once you go the hang of the syntax, you'll find yourself using Fustigate arrays quite oft.

Bash or Python?

Which begs the question: When should yous employ Bash arrays instead of other scripting languages such as Python?

To me, it all boils down to dependencies—if you can solve the problem at mitt using simply calls to command-line tools, you might every bit well use Bash. But for times when your script is role of a larger Python project, you might every bit well employ Python.

For case, we could take turned to Python to implement the parameter sweep, but we would take ended upwards only writing a wrapper around Bash:

                      import subprocess
                          all_threads =              [              1,              two,              4,              8,              sixteen,              32,              64,              128              ]              
              all_runtimes =              [              ]            
                          # Launch pipeline on each number of threads              
              for              t              in              all_threads:
              cmd =              './pipeline --threads {}'.format(t)            
                          # Use the subprocess module to fetch the render output              
              p = subprocess.Popen(cmd,              stdout=subprocess.Piping,              trounce=True)              
              output = p.communicate(              )              [              0              ]              
              all_runtimes.append(output)

Since there's no getting effectually the command line in this example, using Bash directly is preferable.

Time for a shameless plug

This article is based on a talk I gave at OSCON, where I presented the live-coding workshop You Don't Know Bash. No slides, no clickers—merely me and the audition typing away at the control line, exploring the wondrous earth of Bash.

This article originally appeared on Medium and is republished with permission.