Issue
I've created a script using the requests module to scrape content from this webpage. I'm willing to get the content revealed upon pressing the Income Statement and then annual tab. I could fetch them using the requests module and the BeautifulSoup library if a few fields, as in Operating Revenue within Total Revenue were not generated dynamically.
So, I've used the following method to fetch them. Now, the only problem that I can't solve is getting the values under the TTM column. The TTM values are within the keys, starting with trailing in the JSON content. I can't integrate them accordingly so that the results get printed in this manner:
Breakdown TTM 9/29/2023 6/29/2023
This is what I've tried with:
import requests
from pprint import pprint
from bs4 import BeautifulSoup
link = 'https://query1.finance.yahoo.com/ws/fundamentals-timeseries/v1/finance/timeseries/AAPL'
headers = {
'Accept': '*/*',
'Accept-Encoding': 'gzip, deflate, br',
'Accept-Language': 'en-US,en;q=0.9',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36',
'Origin': 'https://finance.yahoo.com',
'Referer': 'https://finance.yahoo.com/quote/AAPL/financials?p=AAPL',
}
params = {
'lang': 'en-US',
'region': 'US',
'symbol': 'AAPL',
'padTimeSeries': 'true',
'type': 'annualTaxEffectOfUnusualItems,trailingTaxEffectOfUnusualItems,annualTaxRateForCalcs,trailingTaxRateForCalcs,annualNormalizedEBITDA,trailingNormalizedEBITDA,annualNormalizedDilutedEPS,trailingNormalizedDilutedEPS,annualNormalizedBasicEPS,trailingNormalizedBasicEPS,annualTotalUnusualItems,trailingTotalUnusualItems,annualTotalUnusualItemsExcludingGoodwill,trailingTotalUnusualItemsExcludingGoodwill,annualNetIncomeFromContinuingOperationNetMinorityInterest,trailingNetIncomeFromContinuingOperationNetMinorityInterest,annualReconciledDepreciation,trailingReconciledDepreciation,annualReconciledCostOfRevenue,trailingReconciledCostOfRevenue,annualEBITDA,trailingEBITDA,annualEBIT,trailingEBIT,annualNetInterestIncome,trailingNetInterestIncome,annualInterestExpense,trailingInterestExpense,annualInterestIncome,trailingInterestIncome,annualContinuingAndDiscontinuedDilutedEPS,trailingContinuingAndDiscontinuedDilutedEPS,annualContinuingAndDiscontinuedBasicEPS,trailingContinuingAndDiscontinuedBasicEPS,annualNormalizedIncome,trailingNormalizedIncome,annualNetIncomeFromContinuingAndDiscontinuedOperation,trailingNetIncomeFromContinuingAndDiscontinuedOperation,annualTotalExpenses,trailingTotalExpenses,annualRentExpenseSupplemental,trailingRentExpenseSupplemental,annualReportedNormalizedDilutedEPS,trailingReportedNormalizedDilutedEPS,annualReportedNormalizedBasicEPS,trailingReportedNormalizedBasicEPS,annualTotalOperatingIncomeAsReported,trailingTotalOperatingIncomeAsReported,annualDividendPerShare,trailingDividendPerShare,annualDilutedAverageShares,trailingDilutedAverageShares,annualBasicAverageShares,trailingBasicAverageShares,annualDilutedEPS,trailingDilutedEPS,annualDilutedEPSOtherGainsLosses,trailingDilutedEPSOtherGainsLosses,annualTaxLossCarryforwardDilutedEPS,trailingTaxLossCarryforwardDilutedEPS,annualDilutedAccountingChange,trailingDilutedAccountingChange,annualDilutedExtraordinary,trailingDilutedExtraordinary,annualDilutedDiscontinuousOperations,trailingDilutedDiscontinuousOperations,annualDilutedContinuousOperations,trailingDilutedContinuousOperations,annualBasicEPS,trailingBasicEPS,annualBasicEPSOtherGainsLosses,trailingBasicEPSOtherGainsLosses,annualTaxLossCarryforwardBasicEPS,trailingTaxLossCarryforwardBasicEPS,annualBasicAccountingChange,trailingBasicAccountingChange,annualBasicExtraordinary,trailingBasicExtraordinary,annualBasicDiscontinuousOperations,trailingBasicDiscontinuousOperations,annualBasicContinuousOperations,trailingBasicContinuousOperations,annualDilutedNIAvailtoComStockholders,trailingDilutedNIAvailtoComStockholders,annualAverageDilutionEarnings,trailingAverageDilutionEarnings,annualNetIncomeCommonStockholders,trailingNetIncomeCommonStockholders,annualOtherunderPreferredStockDividend,trailingOtherunderPreferredStockDividend,annualPreferredStockDividends,trailingPreferredStockDividends,annualNetIncome,trailingNetIncome,annualMinorityInterests,trailingMinorityInterests,annualNetIncomeIncludingNoncontrollingInterests,trailingNetIncomeIncludingNoncontrollingInterests,annualNetIncomeFromTaxLossCarryforward,trailingNetIncomeFromTaxLossCarryforward,annualNetIncomeExtraordinary,trailingNetIncomeExtraordinary,annualNetIncomeDiscontinuousOperations,trailingNetIncomeDiscontinuousOperations,annualNetIncomeContinuousOperations,trailingNetIncomeContinuousOperations,annualEarningsFromEquityInterestNetOfTax,trailingEarningsFromEquityInterestNetOfTax,annualTaxProvision,trailingTaxProvision,annualPretaxIncome,trailingPretaxIncome,annualOtherIncomeExpense,trailingOtherIncomeExpense,annualOtherNonOperatingIncomeExpenses,trailingOtherNonOperatingIncomeExpenses,annualSpecialIncomeCharges,trailingSpecialIncomeCharges,annualGainOnSaleOfPPE,trailingGainOnSaleOfPPE,annualGainOnSaleOfBusiness,trailingGainOnSaleOfBusiness,annualOtherSpecialCharges,trailingOtherSpecialCharges,annualWriteOff,trailingWriteOff,annualImpairmentOfCapitalAssets,trailingImpairmentOfCapitalAssets,annualRestructuringAndMergernAcquisition,trailingRestructuringAndMergernAcquisition,annualSecuritiesAmortization,trailingSecuritiesAmortization,annualEarningsFromEquityInterest,trailingEarningsFromEquityInterest,annualGainOnSaleOfSecurity,trailingGainOnSaleOfSecurity,annualNetNonOperatingInterestIncomeExpense,trailingNetNonOperatingInterestIncomeExpense,annualTotalOtherFinanceCost,trailingTotalOtherFinanceCost,annualInterestExpenseNonOperating,trailingInterestExpenseNonOperating,annualInterestIncomeNonOperating,trailingInterestIncomeNonOperating,annualOperatingIncome,trailingOperatingIncome,annualOperatingExpense,trailingOperatingExpense,annualOtherOperatingExpenses,trailingOtherOperatingExpenses,annualOtherTaxes,trailingOtherTaxes,annualProvisionForDoubtfulAccounts,trailingProvisionForDoubtfulAccounts,annualDepreciationAmortizationDepletionIncomeStatement,trailingDepreciationAmortizationDepletionIncomeStatement,annualDepletionIncomeStatement,trailingDepletionIncomeStatement,annualDepreciationAndAmortizationInIncomeStatement,trailingDepreciationAndAmortizationInIncomeStatement,annualAmortization,trailingAmortization,annualAmortizationOfIntangiblesIncomeStatement,trailingAmortizationOfIntangiblesIncomeStatement,annualDepreciationIncomeStatement,trailingDepreciationIncomeStatement,annualResearchAndDevelopment,trailingResearchAndDevelopment,annualSellingGeneralAndAdministration,trailingSellingGeneralAndAdministration,annualSellingAndMarketingExpense,trailingSellingAndMarketingExpense,annualGeneralAndAdministrativeExpense,trailingGeneralAndAdministrativeExpense,annualOtherGandA,trailingOtherGandA,annualInsuranceAndClaims,trailingInsuranceAndClaims,annualRentAndLandingFees,trailingRentAndLandingFees,annualSalariesAndWages,trailingSalariesAndWages,annualGrossProfit,trailingGrossProfit,annualCostOfRevenue,trailingCostOfRevenue,annualTotalRevenue,trailingTotalRevenue,annualExciseTaxes,trailingExciseTaxes,annualOperatingRevenue,trailingOperatingRevenue',
'merge': 'false',
'period1': '493590046',
'period2': '1705664498',
'corsDomain': 'finance.yahoo.com',
}
res = requests.get(link,params=params,headers=headers)
for item in res.json()['timeseries']['result']:
for i,v in item.items():
if i=="meta" or i=="timestamp": continue
if i.startswith('annual'):
title = i.strip('annual')
try:
third_val_raw = v[2]['reportedValue']['raw']
third_val = int(third_val_raw) if third_val_raw.is_integer() else third_val_raw
except (IndexError,TypeError): third_val = ""
try:
fourth_val_raw = v[3]['reportedValue']['raw']
fourth_val = int(fourth_val_raw) if fourth_val_raw.is_integer() else fourth_val_raw
except (IndexError,TypeError): fourth_val = ""
print(title,fourth_val,third_val)
Current output (truncated) are like (Missing TTM values):
NetIncomeFromContinuingOperationNetMinorityInterest 96995000000 99803000000
TotalOperatingIncomeAsReported 114301000000 119437000000
BasicAverageShares 16215963000
ReconciledDepreciatio 11519000000 11104000000
OperatingExpense 54847000000 51345000000
TotalRevenue 383285000000 394328000000
ResearchAndDevelopment 29915000000 26251000000
InterestExpense 3933000000 2931000000
DilutedNIAvailtoComStockholders 96995000000 99803000000
NetIncomeIncludingNoncontrollingInterests 96995000000 99803000000
How can I fetch the values under the TTM column and print them with other results the script is currently producing?
Solution
IIUC you want to construct the dataframe from the returned data (with TTM column as shown on the page).
from itertools import zip_longest
import numpy as np
import pandas as pd
import requests
from bs4 import BeautifulSoup
link = "https://query1.finance.yahoo.com/ws/fundamentals-timeseries/v1/finance/timeseries/AAPL"
headers = {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate, br",
"Accept-Language": "en-US,en;q=0.9",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36",
"Origin": "https://finance.yahoo.com",
"Referer": "https://finance.yahoo.com/quote/AAPL/financials?p=AAPL",
}
params = {
"lang": "en-US",
"region": "US",
"symbol": "AAPL",
"padTimeSeries": "true",
"type": "annualTaxEffectOfUnusualItems,trailingTaxEffectOfUnusualItems,annualTaxRateForCalcs,trailingTaxRateForCalcs,annualNormalizedEBITDA,trailingNormalizedEBITDA,annualNormalizedDilutedEPS,trailingNormalizedDilutedEPS,annualNormalizedBasicEPS,trailingNormalizedBasicEPS,annualTotalUnusualItems,trailingTotalUnusualItems,annualTotalUnusualItemsExcludingGoodwill,trailingTotalUnusualItemsExcludingGoodwill,annualNetIncomeFromContinuingOperationNetMinorityInterest,trailingNetIncomeFromContinuingOperationNetMinorityInterest,annualReconciledDepreciation,trailingReconciledDepreciation,annualReconciledCostOfRevenue,trailingReconciledCostOfRevenue,annualEBITDA,trailingEBITDA,annualEBIT,trailingEBIT,annualNetInterestIncome,trailingNetInterestIncome,annualInterestExpense,trailingInterestExpense,annualInterestIncome,trailingInterestIncome,annualContinuingAndDiscontinuedDilutedEPS,trailingContinuingAndDiscontinuedDilutedEPS,annualContinuingAndDiscontinuedBasicEPS,trailingContinuingAndDiscontinuedBasicEPS,annualNormalizedIncome,trailingNormalizedIncome,annualNetIncomeFromContinuingAndDiscontinuedOperation,trailingNetIncomeFromContinuingAndDiscontinuedOperation,annualTotalExpenses,trailingTotalExpenses,annualRentExpenseSupplemental,trailingRentExpenseSupplemental,annualReportedNormalizedDilutedEPS,trailingReportedNormalizedDilutedEPS,annualReportedNormalizedBasicEPS,trailingReportedNormalizedBasicEPS,annualTotalOperatingIncomeAsReported,trailingTotalOperatingIncomeAsReported,annualDividendPerShare,trailingDividendPerShare,annualDilutedAverageShares,trailingDilutedAverageShares,annualBasicAverageShares,trailingBasicAverageShares,annualDilutedEPS,trailingDilutedEPS,annualDilutedEPSOtherGainsLosses,trailingDilutedEPSOtherGainsLosses,annualTaxLossCarryforwardDilutedEPS,trailingTaxLossCarryforwardDilutedEPS,annualDilutedAccountingChange,trailingDilutedAccountingChange,annualDilutedExtraordinary,trailingDilutedExtraordinary,annualDilutedDiscontinuousOperations,trailingDilutedDiscontinuousOperations,annualDilutedContinuousOperations,trailingDilutedContinuousOperations,annualBasicEPS,trailingBasicEPS,annualBasicEPSOtherGainsLosses,trailingBasicEPSOtherGainsLosses,annualTaxLossCarryforwardBasicEPS,trailingTaxLossCarryforwardBasicEPS,annualBasicAccountingChange,trailingBasicAccountingChange,annualBasicExtraordinary,trailingBasicExtraordinary,annualBasicDiscontinuousOperations,trailingBasicDiscontinuousOperations,annualBasicContinuousOperations,trailingBasicContinuousOperations,annualDilutedNIAvailtoComStockholders,trailingDilutedNIAvailtoComStockholders,annualAverageDilutionEarnings,trailingAverageDilutionEarnings,annualNetIncomeCommonStockholders,trailingNetIncomeCommonStockholders,annualOtherunderPreferredStockDividend,trailingOtherunderPreferredStockDividend,annualPreferredStockDividends,trailingPreferredStockDividends,annualNetIncome,trailingNetIncome,annualMinorityInterests,trailingMinorityInterests,annualNetIncomeIncludingNoncontrollingInterests,trailingNetIncomeIncludingNoncontrollingInterests,annualNetIncomeFromTaxLossCarryforward,trailingNetIncomeFromTaxLossCarryforward,annualNetIncomeExtraordinary,trailingNetIncomeExtraordinary,annualNetIncomeDiscontinuousOperations,trailingNetIncomeDiscontinuousOperations,annualNetIncomeContinuousOperations,trailingNetIncomeContinuousOperations,annualEarningsFromEquityInterestNetOfTax,trailingEarningsFromEquityInterestNetOfTax,annualTaxProvision,trailingTaxProvision,annualPretaxIncome,trailingPretaxIncome,annualOtherIncomeExpense,trailingOtherIncomeExpense,annualOtherNonOperatingIncomeExpenses,trailingOtherNonOperatingIncomeExpenses,annualSpecialIncomeCharges,trailingSpecialIncomeCharges,annualGainOnSaleOfPPE,trailingGainOnSaleOfPPE,annualGainOnSaleOfBusiness,trailingGainOnSaleOfBusiness,annualOtherSpecialCharges,trailingOtherSpecialCharges,annualWriteOff,trailingWriteOff,annualImpairmentOfCapitalAssets,trailingImpairmentOfCapitalAssets,annualRestructuringAndMergernAcquisition,trailingRestructuringAndMergernAcquisition,annualSecuritiesAmortization,trailingSecuritiesAmortization,annualEarningsFromEquityInterest,trailingEarningsFromEquityInterest,annualGainOnSaleOfSecurity,trailingGainOnSaleOfSecurity,annualNetNonOperatingInterestIncomeExpense,trailingNetNonOperatingInterestIncomeExpense,annualTotalOtherFinanceCost,trailingTotalOtherFinanceCost,annualInterestExpenseNonOperating,trailingInterestExpenseNonOperating,annualInterestIncomeNonOperating,trailingInterestIncomeNonOperating,annualOperatingIncome,trailingOperatingIncome,annualOperatingExpense,trailingOperatingExpense,annualOtherOperatingExpenses,trailingOtherOperatingExpenses,annualOtherTaxes,trailingOtherTaxes,annualProvisionForDoubtfulAccounts,trailingProvisionForDoubtfulAccounts,annualDepreciationAmortizationDepletionIncomeStatement,trailingDepreciationAmortizationDepletionIncomeStatement,annualDepletionIncomeStatement,trailingDepletionIncomeStatement,annualDepreciationAndAmortizationInIncomeStatement,trailingDepreciationAndAmortizationInIncomeStatement,annualAmortization,trailingAmortization,annualAmortizationOfIntangiblesIncomeStatement,trailingAmortizationOfIntangiblesIncomeStatement,annualDepreciationIncomeStatement,trailingDepreciationIncomeStatement,annualResearchAndDevelopment,trailingResearchAndDevelopment,annualSellingGeneralAndAdministration,trailingSellingGeneralAndAdministration,annualSellingAndMarketingExpense,trailingSellingAndMarketingExpense,annualGeneralAndAdministrativeExpense,trailingGeneralAndAdministrativeExpense,annualOtherGandA,trailingOtherGandA,annualInsuranceAndClaims,trailingInsuranceAndClaims,annualRentAndLandingFees,trailingRentAndLandingFees,annualSalariesAndWages,trailingSalariesAndWages,annualGrossProfit,trailingGrossProfit,annualCostOfRevenue,trailingCostOfRevenue,annualTotalRevenue,trailingTotalRevenue,annualExciseTaxes,trailingExciseTaxes,annualOperatingRevenue,trailingOperatingRevenue",
"merge": "false",
"period1": "493590046",
"period2": "1705664498",
"corsDomain": "finance.yahoo.com",
}
res = requests.get(link, params=params, headers=headers).json()
all_data = []
for r in res["timeseries"]["result"]:
type_ = r["meta"]["type"][0]
if "timestamp" not in r:
continue
if type_.startswith("trailing"):
df = pd.DataFrame(
{"TTM": [r[type_][0]["reportedValue"]["raw"]]},
index=[type_.replace("trailing", "annual")],
dtype=float,
)
else:
df = pd.DataFrame(
{
t: v["reportedValue"]["raw"]
for t, v in zip_longest(
r["timestamp"],
(t for t in r[type_] if t),
fillvalue={"reportedValue": {"raw": np.nan}},
)
},
index=[type_],
dtype=float,
)
all_data.append(df)
out = (
pd.concat(all_data)
.groupby(level=0, as_index=False, group_keys=False)
.apply(lambda g: g.ffill().bfill().drop_duplicates())
)
out.columns = [
c if c == "TTM" else pd.to_datetime(c, unit="s").date() for c in out.columns
]
pd.options.display.float_format = "{:.2f}".format
print(out[out.columns[::-1]])
Prints:
TTM 2023-09-30 2022-09-30 2021-09-30 2020-09-30
annualBasicAverageShares 15744231000.00 NaN 16215963000.00 16701272000.00 17352119000.00
annualBasicEPS 6.16 NaN 6.15 5.67 3.31
annualCostOfRevenue 214137000000.00 214137000000.00 223546000000.00 212981000000.00 169559000000.00
annualDilutedAverageShares 15812547000.00 NaN 16325819000.00 16864919000.00 17528214000.00
annualDilutedEPS 6.13 NaN 6.11 5.61 3.28
annualDilutedNIAvailtoComStockholders 96995000000.00 96995000000.00 99803000000.00 94680000000.00 57411000000.00
annualEBIT 117669000000.00 117669000000.00 122034000000.00 111852000000.00 69964000000.00
annualEBITDA 129188000000.00 129188000000.00 133138000000.00 123136000000.00 81020000000.00
annualGrossProfit 169148000000.00 169148000000.00 170782000000.00 152836000000.00 104956000000.00
annualInterestExpense 3933000000.00 3933000000.00 2931000000.00 2645000000.00 2873000000.00
annualInterestExpenseNonOperating 3933000000.00 3933000000.00 2931000000.00 2645000000.00 2873000000.00
annualInterestIncome 3750000000.00 3750000000.00 2825000000.00 2843000000.00 3763000000.00
annualInterestIncomeNonOperating 3750000000.00 3750000000.00 2825000000.00 2843000000.00 3763000000.00
annualNetIncome 96995000000.00 96995000000.00 99803000000.00 94680000000.00 57411000000.00
annualNetIncomeCommonStockholders 96995000000.00 96995000000.00 99803000000.00 94680000000.00 57411000000.00
annualNetIncomeContinuousOperations 96995000000.00 96995000000.00 99803000000.00 94680000000.00 57411000000.00
annualNetIncomeFromContinuingAndDiscontinuedOperation 96995000000.00 96995000000.00 99803000000.00 94680000000.00 57411000000.00
annualNetIncomeFromContinuingOperationNetMinorityInterest 96995000000.00 96995000000.00 99803000000.00 94680000000.00 57411000000.00
annualNetIncomeIncludingNoncontrollingInterests 96995000000.00 96995000000.00 99803000000.00 94680000000.00 57411000000.00
annualNetInterestIncome -183000000.00 -183000000.00 -106000000.00 198000000.00 890000000.00
annualNetNonOperatingInterestIncomeExpense -183000000.00 -183000000.00 -106000000.00 198000000.00 890000000.00
annualNormalizedEBITDA 129188000000.00 129188000000.00 133138000000.00 123136000000.00 81020000000.00
annualNormalizedIncome 96995000000.00 96995000000.00 99803000000.00 94680000000.00 57411000000.00
annualOperatingExpense 54847000000.00 54847000000.00 51345000000.00 43887000000.00 38668000000.00
annualOperatingIncome 114301000000.00 114301000000.00 119437000000.00 108949000000.00 66288000000.00
annualOperatingRevenue 383285000000.00 383285000000.00 394328000000.00 365817000000.00 274515000000.00
annualOtherIncomeExpense -382000000.00 -382000000.00 -228000000.00 60000000.00 -87000000.00
annualOtherNonOperatingIncomeExpenses -382000000.00 -382000000.00 -228000000.00 60000000.00 -87000000.00
annualPretaxIncome 113736000000.00 113736000000.00 119103000000.00 109207000000.00 67091000000.00
annualReconciledCostOfRevenue 214137000000.00 214137000000.00 223546000000.00 212981000000.00 169559000000.00
annualReconciledDepreciation 11519000000.00 11519000000.00 11104000000.00 11284000000.00 11056000000.00
annualResearchAndDevelopment 29915000000.00 29915000000.00 26251000000.00 21914000000.00 18752000000.00
annualSellingGeneralAndAdministration 24932000000.00 24932000000.00 25094000000.00 21973000000.00 19916000000.00
annualTaxEffectOfUnusualItems 0.00 0.00 0.00 0.00 0.00
annualTaxProvision 16741000000.00 16741000000.00 19300000000.00 14527000000.00 9680000000.00
annualTaxRateForCalcs 0.15 0.15 0.16 0.13 0.14
annualTotalExpenses 268984000000.00 268984000000.00 274891000000.00 256868000000.00 208227000000.00
annualTotalOperatingIncomeAsReported 114301000000.00 114301000000.00 119437000000.00 108949000000.00 66288000000.00
annualTotalRevenue 383285000000.00 383285000000.00 394328000000.00 365817000000.00 274515000000.00
Answered By - Andrej Kesely
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.