There are a few options here - I can use a third party ETL tool, such as Scribe, to load the Excel file into CRM directly, but this can be an overkill. Or I can re-do the whole data import, but this may not be viable because data have been updated/deleted since the first import. So I decided to use the export-for-reimport approach.
The environment that I worked with is CRM 2016 Online. In earlier CRM versions, the Excel export record limit is 10,000. To export more than that, you will have to rely on registry key tweak or tools to increase the limit. But now the limit is increased to 100,000 - such a life saver. Also, the option to specify for a re-import is taken away. I take it in the new CRM, any export to Excel, by default record guid is included and hidden.
So I exported all data with Phone # columns that I wanted to update into Excel (let's name this Destination worksheet). In the Excel, I attached another Excel Worksheet with updated Phone #s (let's name this Source worksheet). From there, all I needed to do is retrieve and match the updated value from the source into the destination.
This is the exported Excel worksheet (destination)

And this is the source worksheet with updated phone #s (source)

So the formula is VLOOKUP (lookup_value, table_array, col_index_num, [range_lookup])
In the exported Excel example (destination), in the Main Phone cell, used formula: VLOOKUP($A2,Sheet1!$A$2:$C$780,2,0), where
$A2 = the field that I wanted to look up against, in this case, Full name
Sheet1!$A$2:$C$2000 = the range to search (in the source)
2 = the returned column index, 2 indicated the second column, which is Column B
0 = always set to 0 for an exact match, if skipped or set to 1, you will get weird result.
After populating your exported Excel, re-import it, and you will have your updated info.