Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
7.585931234s trace:
https://jaeger.dev.storj.io/trace/20a7eed07221a46a
FAIL (2025-12-13 05:36:12 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 13 05:33:58.098067468 (182h4m57.969150503s) [upload=159506]: Starting upload
Dec 13 05:33:58.098123904 (182h4m57.969206949s) [upload=159506]: Done starting upload
Dec 13 05:33:58.098354594 (182h4m57.969437639s) [upload=159506]: Uploading segments...
Dec 13 05:33:58.098654204 (182h4m57.969737239s) [upload=159506]: Got next segment. Inline: false
Dec 13 05:33:58.098692758 (182h4m57.969775793s) [upload=159506]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 13 05:33:58.098697167 (182h4m57.969780202s) [upload=159506]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 13 05:33:58.362129190 (182h4m58.233212245s) [seg_pos={0 0}, upload=159506]: Begin upload segment...
Dec 13 05:33:58.362185707 (182h4m58.233268742s) [seg_pos={0 0}, upload=159506]: Done begin upload segment.
Dec 13 05:33:58.362216586 (182h4m58.233299631s) [noise=true , address=77.90.31.11:4555 , piece_id=DDDWRVUGDOM6ALDY5K2MWEO2HR62GHFJF5BIWDTUALHHXZRXB4CQ, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.362268716 (182h4m58.233351761s) [noise=true , address=77.90.31.10:4520 , piece_id=PTDKHO4QVVWVWCM3TPVNPVSWT7I267WHYI6PZVRILOPH423K2NZQ, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.362778736 (182h4m58.233861771s) [noise=true , address=77.90.31.14:4663 , piece_id=HYNK74E4LI2IRX7YV6YDC5IPINRHXKCCW6GQXZT4ZORGSGM6DURA, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.363444535 (182h4m58.23452758s) [noise=true , address=145.79.227.50:4373 , piece_id=HDJ3F7PSHTMXGMKKCZFQ2EVCWDBZGJ5W3I2EOMN46ZBFJNRLVJDQ, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.363518375 (182h4m58.23460141s) [noise=true , address=77.90.31.16:4734 , piece_id=4VLPNEXKROFP2QXREKBDMKESFGZUU45CTOCUHJ6EWRLCGNGNVO2A, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.363841891 (182h4m58.234924936s) [noise=true , address=145.79.227.40:4049 , piece_id=PASZUKH76WKE62G3UBUXEH32TPF26CS7HQQDKN5UZ5RICWQFQNFA, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.363858633 (182h4m58.234941669s) [noise=true , address=145.79.227.32:4836 , piece_id=DBZUHMNUAVAC6UZUDUOTK4WBLV5C3CNIPC25FJBNWPQOTV2IAX4A, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.364176749 (182h4m58.235259794s) [noise=true , address=77.90.31.12:4589 , piece_id=TEEE5AM3GVE6QA5XDCT6NEQUSFJGSWRIW4DBFYNXFUEQRXQFI5UA, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.364264025 (182h4m58.23534707s) [noise=true , address=145.79.227.50:4374 , piece_id=2TGRU7HF5SROJ4EEHG32HI2EKAHH4GQL4XXYWMNGSATCYLCNVUFQ, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.364451392 (182h4m58.235534427s) [noise=true , address=145.79.227.46:4603 , piece_id=WTP7JT4BFDJVR7JHFLC5OIPPJ3CZSCHRCHQDP67QODHH3BJ2GB4Q, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.364739200 (182h4m58.235822235s) [noise=true , address=145.79.227.36:4519 , piece_id=BHHN6RRNKPY57DNH44ILZVXAWL66IHR2A5YQ4DSBE22LQFPZSBDQ, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.365064920 (182h4m58.236147965s) [noise=true , address=145.79.227.22:4509 , piece_id=BDZIDW725LYFQF4CZKMOGMTKMSREVVMUQBTIQH5IC43EOHGFJD4A, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.365389158 (182h4m58.236472203s) [noise=true , address=145.79.227.48:4303 , piece_id=XWN2P6JSSOLG6XTQLNQVF3RHHXJFRIQKDLYDXWOV7UFPV5SK32OQ, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.365680051 (182h4m58.236763087s) [noise=true , address=145.79.227.48:4302 , piece_id=CDQGUACHANDREDEST3QCZ44CDWL3QARYT4B6EF4QGD4SOPQHETHA, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.365963611 (182h4m58.237046646s) [noise=true , address=145.79.227.42:4123 , piece_id=XMJDDBGGY44FRQF73PSULH5L37MBCNWBJXT62PHX7IKSVJ44S24Q, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.366485655 (182h4m58.23756869s) [noise=true , address=145.79.227.30:4763 , piece_id=CAEOGXNNUYUKEVKQMAMI4NTDPTIG67H4TELFXIKJE3SFNGQ4NXJQ, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:59.473265431 (182h4m59.344348466s) [noise=true , address=77.90.31.16:4734 , piece_id=4VLPNEXKROFP2QXREKBDMKESFGZUU45CTOCUHJ6EWRLCGNGNVO2A, seg_pos={0 0}, upload=159506]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000
Dec 13 05:33:59.786446836 (182h4m59.657529891s) [noise=true , address=145.79.227.30:4763 , piece_id=CAEOGXNNUYUKEVKQMAMI4NTDPTIG67H4TELFXIKJE3SFNGQ4NXJQ, seg_pos={0 0}, upload=159506]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000
Dec 13 05:34:00.203382529 (182h5m0.074465564s) [upload=159506]: Next returned nil segment
Dec 13 05:34:00.203398039 (182h5m0.074481084s) [upload=159506]: Waiting for error group managing segments...
Dec 13 05:34:00.521187459 (182h5m0.392270494s) [noise=true , address=145.79.227.40:4049 , piece_id=PASZUKH76WKE62G3UBUXEH32TPF26CS7HQQDKN5UZ5RICWQFQNFA, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:00.521787982 (182h5m0.392871017s) [noise=true , address=145.79.227.48:4302 , piece_id=CDQGUACHANDREDEST3QCZ44CDWL3QARYT4B6EF4QGD4SOPQHETHA, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:00.525174060 (182h5m0.396257105s) [noise=true , address=145.79.227.22:4509 , piece_id=BDZIDW725LYFQF4CZKMOGMTKMSREVVMUQBTIQH5IC43EOHGFJD4A, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:00.529552096 (182h5m0.400635131s) [noise=true , address=145.79.227.42:4123 , piece_id=XMJDDBGGY44FRQF73PSULH5L37MBCNWBJXT62PHX7IKSVJ44S24Q, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:00.542433055 (182h5m0.41351609s) [noise=true , address=77.90.31.10:4520 , piece_id=PTDKHO4QVVWVWCM3TPVNPVSWT7I267WHYI6PZVRILOPH423K2NZQ, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:00.552483503 (182h5m0.423566548s) [noise=true , address=145.79.227.32:4836 , piece_id=DBZUHMNUAVAC6UZUDUOTK4WBLV5C3CNIPC25FJBNWPQOTV2IAX4A, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:00.612273003 (182h5m0.483356048s) [noise=true , address=77.90.31.14:4663 , piece_id=HYNK74E4LI2IRX7YV6YDC5IPINRHXKCCW6GQXZT4ZORGSGM6DURA, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:00.835989661 (182h5m0.707072706s) [noise=true , address=145.79.227.50:4373 , piece_id=HDJ3F7PSHTMXGMKKCZFQ2EVCWDBZGJ5W3I2EOMN46ZBFJNRLVJDQ, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:01.196293326 (182h5m1.067376371s) [noise=true , address=145.79.227.50:4374 , piece_id=2TGRU7HF5SROJ4EEHG32HI2EKAHH4GQL4XXYWMNGSATCYLCNVUFQ, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:01.226460089 (182h5m1.097543135s) [noise=true , address=145.79.227.36:4519 , piece_id=BHHN6RRNKPY57DNH44ILZVXAWL66IHR2A5YQ4DSBE22LQFPZSBDQ, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:01.265467076 (182h5m1.136550121s) [noise=true , address=145.79.227.48:4303 , piece_id=XWN2P6JSSOLG6XTQLNQVF3RHHXJFRIQKDLYDXWOV7UFPV5SK32OQ, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:01.509373308 (182h5m1.380456353s) [noise=true , address=145.79.227.46:4603 , piece_id=WTP7JT4BFDJVR7JHFLC5OIPPJ3CZSCHRCHQDP67QODHH3BJ2GB4Q, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:02.844300840 (182h5m2.715383885s) [noise=true , address=77.90.31.12:4589 , piece_id=TEEE5AM3GVE6QA5XDCT6NEQUSFJGSWRIW4DBFYNXFUEQRXQFI5UA, seg_pos={0 0}, upload=159506]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 13 05:34:24.040889140 (182h5m23.911972185s) [noise=true , address=77.90.31.11:4555 , piece_id=DDDWRVUGDOM6ALDY5K2MWEO2HR62GHFJF5BIWDTUALHHXZRXB4CQ, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:24.040917073 (182h5m23.912000118s) [seg_pos={0 0}, upload=159506]: Segment reached optimal threshold of 13 pieces.
Dec 13 05:34:24.041085103 (182h5m23.912168148s) [seg_pos={0 0}, upload=159506]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 13 05:34:24.041168802 (182h5m23.912251847s) [upload=159506]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 13 05:34:24.041177479 (182h5m23.912260524s) [upload=159506]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 13 05:34:24.348509366 (182h5m24.219592411s) [upload=159506]: Done uploading segments...
Dec 13 05:34:24.348533872 (182h5m24.219616907s) [upload=159506]: Upload finished. err: <nil>
Duration: 7.585959959s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
7.701610619s trace:
https://jaeger.dev.storj.io/trace/20a7eed07221a46a
FAIL (2025-12-13 05:35:55 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 13 05:33:58.098067468 (182h4m57.969150503s) [upload=159506]: Starting upload
Dec 13 05:33:58.098123904 (182h4m57.969206949s) [upload=159506]: Done starting upload
Dec 13 05:33:58.098354594 (182h4m57.969437639s) [upload=159506]: Uploading segments...
Dec 13 05:33:58.098654204 (182h4m57.969737239s) [upload=159506]: Got next segment. Inline: false
Dec 13 05:33:58.098692758 (182h4m57.969775793s) [upload=159506]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 13 05:33:58.098697167 (182h4m57.969780202s) [upload=159506]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 13 05:33:58.362129190 (182h4m58.233212245s) [seg_pos={0 0}, upload=159506]: Begin upload segment...
Dec 13 05:33:58.362185707 (182h4m58.233268742s) [seg_pos={0 0}, upload=159506]: Done begin upload segment.
Dec 13 05:33:58.362216586 (182h4m58.233299631s) [noise=true , address=77.90.31.11:4555 , piece_id=DDDWRVUGDOM6ALDY5K2MWEO2HR62GHFJF5BIWDTUALHHXZRXB4CQ, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.362268716 (182h4m58.233351761s) [noise=true , address=77.90.31.10:4520 , piece_id=PTDKHO4QVVWVWCM3TPVNPVSWT7I267WHYI6PZVRILOPH423K2NZQ, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.362778736 (182h4m58.233861771s) [noise=true , address=77.90.31.14:4663 , piece_id=HYNK74E4LI2IRX7YV6YDC5IPINRHXKCCW6GQXZT4ZORGSGM6DURA, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.363444535 (182h4m58.23452758s) [noise=true , address=145.79.227.50:4373 , piece_id=HDJ3F7PSHTMXGMKKCZFQ2EVCWDBZGJ5W3I2EOMN46ZBFJNRLVJDQ, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.363518375 (182h4m58.23460141s) [noise=true , address=77.90.31.16:4734 , piece_id=4VLPNEXKROFP2QXREKBDMKESFGZUU45CTOCUHJ6EWRLCGNGNVO2A, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.363841891 (182h4m58.234924936s) [noise=true , address=145.79.227.40:4049 , piece_id=PASZUKH76WKE62G3UBUXEH32TPF26CS7HQQDKN5UZ5RICWQFQNFA, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.363858633 (182h4m58.234941669s) [noise=true , address=145.79.227.32:4836 , piece_id=DBZUHMNUAVAC6UZUDUOTK4WBLV5C3CNIPC25FJBNWPQOTV2IAX4A, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.364176749 (182h4m58.235259794s) [noise=true , address=77.90.31.12:4589 , piece_id=TEEE5AM3GVE6QA5XDCT6NEQUSFJGSWRIW4DBFYNXFUEQRXQFI5UA, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.364264025 (182h4m58.23534707s) [noise=true , address=145.79.227.50:4374 , piece_id=2TGRU7HF5SROJ4EEHG32HI2EKAHH4GQL4XXYWMNGSATCYLCNVUFQ, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.364451392 (182h4m58.235534427s) [noise=true , address=145.79.227.46:4603 , piece_id=WTP7JT4BFDJVR7JHFLC5OIPPJ3CZSCHRCHQDP67QODHH3BJ2GB4Q, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.364739200 (182h4m58.235822235s) [noise=true , address=145.79.227.36:4519 , piece_id=BHHN6RRNKPY57DNH44ILZVXAWL66IHR2A5YQ4DSBE22LQFPZSBDQ, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.365064920 (182h4m58.236147965s) [noise=true , address=145.79.227.22:4509 , piece_id=BDZIDW725LYFQF4CZKMOGMTKMSREVVMUQBTIQH5IC43EOHGFJD4A, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.365389158 (182h4m58.236472203s) [noise=true , address=145.79.227.48:4303 , piece_id=XWN2P6JSSOLG6XTQLNQVF3RHHXJFRIQKDLYDXWOV7UFPV5SK32OQ, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.365680051 (182h4m58.236763087s) [noise=true , address=145.79.227.48:4302 , piece_id=CDQGUACHANDREDEST3QCZ44CDWL3QARYT4B6EF4QGD4SOPQHETHA, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.365963611 (182h4m58.237046646s) [noise=true , address=145.79.227.42:4123 , piece_id=XMJDDBGGY44FRQF73PSULH5L37MBCNWBJXT62PHX7IKSVJ44S24Q, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.366485655 (182h4m58.23756869s) [noise=true , address=145.79.227.30:4763 , piece_id=CAEOGXNNUYUKEVKQMAMI4NTDPTIG67H4TELFXIKJE3SFNGQ4NXJQ, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:59.473265431 (182h4m59.344348466s) [noise=true , address=77.90.31.16:4734 , piece_id=4VLPNEXKROFP2QXREKBDMKESFGZUU45CTOCUHJ6EWRLCGNGNVO2A, seg_pos={0 0}, upload=159506]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000
Dec 13 05:33:59.786446836 (182h4m59.657529891s) [noise=true , address=145.79.227.30:4763 , piece_id=CAEOGXNNUYUKEVKQMAMI4NTDPTIG67H4TELFXIKJE3SFNGQ4NXJQ, seg_pos={0 0}, upload=159506]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000
Dec 13 05:34:00.203382529 (182h5m0.074465564s) [upload=159506]: Next returned nil segment
Dec 13 05:34:00.203398039 (182h5m0.074481084s) [upload=159506]: Waiting for error group managing segments...
Dec 13 05:34:00.521187459 (182h5m0.392270494s) [noise=true , address=145.79.227.40:4049 , piece_id=PASZUKH76WKE62G3UBUXEH32TPF26CS7HQQDKN5UZ5RICWQFQNFA, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:00.521787982 (182h5m0.392871017s) [noise=true , address=145.79.227.48:4302 , piece_id=CDQGUACHANDREDEST3QCZ44CDWL3QARYT4B6EF4QGD4SOPQHETHA, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:00.525174060 (182h5m0.396257105s) [noise=true , address=145.79.227.22:4509 , piece_id=BDZIDW725LYFQF4CZKMOGMTKMSREVVMUQBTIQH5IC43EOHGFJD4A, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:00.529552096 (182h5m0.400635131s) [noise=true , address=145.79.227.42:4123 , piece_id=XMJDDBGGY44FRQF73PSULH5L37MBCNWBJXT62PHX7IKSVJ44S24Q, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:00.542433055 (182h5m0.41351609s) [noise=true , address=77.90.31.10:4520 , piece_id=PTDKHO4QVVWVWCM3TPVNPVSWT7I267WHYI6PZVRILOPH423K2NZQ, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:00.552483503 (182h5m0.423566548s) [noise=true , address=145.79.227.32:4836 , piece_id=DBZUHMNUAVAC6UZUDUOTK4WBLV5C3CNIPC25FJBNWPQOTV2IAX4A, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:00.612273003 (182h5m0.483356048s) [noise=true , address=77.90.31.14:4663 , piece_id=HYNK74E4LI2IRX7YV6YDC5IPINRHXKCCW6GQXZT4ZORGSGM6DURA, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:00.835989661 (182h5m0.707072706s) [noise=true , address=145.79.227.50:4373 , piece_id=HDJ3F7PSHTMXGMKKCZFQ2EVCWDBZGJ5W3I2EOMN46ZBFJNRLVJDQ, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:01.196293326 (182h5m1.067376371s) [noise=true , address=145.79.227.50:4374 , piece_id=2TGRU7HF5SROJ4EEHG32HI2EKAHH4GQL4XXYWMNGSATCYLCNVUFQ, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:01.226460089 (182h5m1.097543135s) [noise=true , address=145.79.227.36:4519 , piece_id=BHHN6RRNKPY57DNH44ILZVXAWL66IHR2A5YQ4DSBE22LQFPZSBDQ, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:01.265467076 (182h5m1.136550121s) [noise=true , address=145.79.227.48:4303 , piece_id=XWN2P6JSSOLG6XTQLNQVF3RHHXJFRIQKDLYDXWOV7UFPV5SK32OQ, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:01.509373308 (182h5m1.380456353s) [noise=true , address=145.79.227.46:4603 , piece_id=WTP7JT4BFDJVR7JHFLC5OIPPJ3CZSCHRCHQDP67QODHH3BJ2GB4Q, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:02.844300840 (182h5m2.715383885s) [noise=true , address=77.90.31.12:4589 , piece_id=TEEE5AM3GVE6QA5XDCT6NEQUSFJGSWRIW4DBFYNXFUEQRXQFI5UA, seg_pos={0 0}, upload=159506]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 13 05:34:24.040889140 (182h5m23.911972185s) [noise=true , address=77.90.31.11:4555 , piece_id=DDDWRVUGDOM6ALDY5K2MWEO2HR62GHFJF5BIWDTUALHHXZRXB4CQ, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:24.040917073 (182h5m23.912000118s) [seg_pos={0 0}, upload=159506]: Segment reached optimal threshold of 13 pieces.
Dec 13 05:34:24.041085103 (182h5m23.912168148s) [seg_pos={0 0}, upload=159506]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 13 05:34:24.041168802 (182h5m23.912251847s) [upload=159506]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 13 05:34:24.041177479 (182h5m23.912260524s) [upload=159506]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 13 05:34:24.348509366 (182h5m24.219592411s) [upload=159506]: Done uploading segments...
Dec 13 05:34:24.348533872 (182h5m24.219616907s) [upload=159506]: Upload finished. err: <nil>
Duration: 7.701629567s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
8.690422503s trace:
https://jaeger.dev.storj.io/trace/20a7eed07221a46a
FAIL (2025-12-13 05:34:58 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 13 05:33:58.098067468 (182h4m57.969150503s) [upload=159506]: Starting upload
Dec 13 05:33:58.098123904 (182h4m57.969206949s) [upload=159506]: Done starting upload
Dec 13 05:33:58.098354594 (182h4m57.969437639s) [upload=159506]: Uploading segments...
Dec 13 05:33:58.098654204 (182h4m57.969737239s) [upload=159506]: Got next segment. Inline: false
Dec 13 05:33:58.098692758 (182h4m57.969775793s) [upload=159506]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 13 05:33:58.098697167 (182h4m57.969780202s) [upload=159506]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 13 05:33:58.362129190 (182h4m58.233212245s) [seg_pos={0 0}, upload=159506]: Begin upload segment...
Dec 13 05:33:58.362185707 (182h4m58.233268742s) [seg_pos={0 0}, upload=159506]: Done begin upload segment.
Dec 13 05:33:58.362216586 (182h4m58.233299631s) [noise=true , address=77.90.31.11:4555 , piece_id=DDDWRVUGDOM6ALDY5K2MWEO2HR62GHFJF5BIWDTUALHHXZRXB4CQ, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.362268716 (182h4m58.233351761s) [noise=true , address=77.90.31.10:4520 , piece_id=PTDKHO4QVVWVWCM3TPVNPVSWT7I267WHYI6PZVRILOPH423K2NZQ, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.362778736 (182h4m58.233861771s) [noise=true , address=77.90.31.14:4663 , piece_id=HYNK74E4LI2IRX7YV6YDC5IPINRHXKCCW6GQXZT4ZORGSGM6DURA, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.363444535 (182h4m58.23452758s) [noise=true , address=145.79.227.50:4373 , piece_id=HDJ3F7PSHTMXGMKKCZFQ2EVCWDBZGJ5W3I2EOMN46ZBFJNRLVJDQ, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.363518375 (182h4m58.23460141s) [noise=true , address=77.90.31.16:4734 , piece_id=4VLPNEXKROFP2QXREKBDMKESFGZUU45CTOCUHJ6EWRLCGNGNVO2A, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.363841891 (182h4m58.234924936s) [noise=true , address=145.79.227.40:4049 , piece_id=PASZUKH76WKE62G3UBUXEH32TPF26CS7HQQDKN5UZ5RICWQFQNFA, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.363858633 (182h4m58.234941669s) [noise=true , address=145.79.227.32:4836 , piece_id=DBZUHMNUAVAC6UZUDUOTK4WBLV5C3CNIPC25FJBNWPQOTV2IAX4A, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.364176749 (182h4m58.235259794s) [noise=true , address=77.90.31.12:4589 , piece_id=TEEE5AM3GVE6QA5XDCT6NEQUSFJGSWRIW4DBFYNXFUEQRXQFI5UA, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.364264025 (182h4m58.23534707s) [noise=true , address=145.79.227.50:4374 , piece_id=2TGRU7HF5SROJ4EEHG32HI2EKAHH4GQL4XXYWMNGSATCYLCNVUFQ, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.364451392 (182h4m58.235534427s) [noise=true , address=145.79.227.46:4603 , piece_id=WTP7JT4BFDJVR7JHFLC5OIPPJ3CZSCHRCHQDP67QODHH3BJ2GB4Q, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.364739200 (182h4m58.235822235s) [noise=true , address=145.79.227.36:4519 , piece_id=BHHN6RRNKPY57DNH44ILZVXAWL66IHR2A5YQ4DSBE22LQFPZSBDQ, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.365064920 (182h4m58.236147965s) [noise=true , address=145.79.227.22:4509 , piece_id=BDZIDW725LYFQF4CZKMOGMTKMSREVVMUQBTIQH5IC43EOHGFJD4A, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.365389158 (182h4m58.236472203s) [noise=true , address=145.79.227.48:4303 , piece_id=XWN2P6JSSOLG6XTQLNQVF3RHHXJFRIQKDLYDXWOV7UFPV5SK32OQ, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.365680051 (182h4m58.236763087s) [noise=true , address=145.79.227.48:4302 , piece_id=CDQGUACHANDREDEST3QCZ44CDWL3QARYT4B6EF4QGD4SOPQHETHA, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.365963611 (182h4m58.237046646s) [noise=true , address=145.79.227.42:4123 , piece_id=XMJDDBGGY44FRQF73PSULH5L37MBCNWBJXT62PHX7IKSVJ44S24Q, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.366485655 (182h4m58.23756869s) [noise=true , address=145.79.227.30:4763 , piece_id=CAEOGXNNUYUKEVKQMAMI4NTDPTIG67H4TELFXIKJE3SFNGQ4NXJQ, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:59.473265431 (182h4m59.344348466s) [noise=true , address=77.90.31.16:4734 , piece_id=4VLPNEXKROFP2QXREKBDMKESFGZUU45CTOCUHJ6EWRLCGNGNVO2A, seg_pos={0 0}, upload=159506]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000
Dec 13 05:33:59.786446836 (182h4m59.657529891s) [noise=true , address=145.79.227.30:4763 , piece_id=CAEOGXNNUYUKEVKQMAMI4NTDPTIG67H4TELFXIKJE3SFNGQ4NXJQ, seg_pos={0 0}, upload=159506]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000
Dec 13 05:34:00.203382529 (182h5m0.074465564s) [upload=159506]: Next returned nil segment
Dec 13 05:34:00.203398039 (182h5m0.074481084s) [upload=159506]: Waiting for error group managing segments...
Dec 13 05:34:00.521187459 (182h5m0.392270494s) [noise=true , address=145.79.227.40:4049 , piece_id=PASZUKH76WKE62G3UBUXEH32TPF26CS7HQQDKN5UZ5RICWQFQNFA, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:00.521787982 (182h5m0.392871017s) [noise=true , address=145.79.227.48:4302 , piece_id=CDQGUACHANDREDEST3QCZ44CDWL3QARYT4B6EF4QGD4SOPQHETHA, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:00.525174060 (182h5m0.396257105s) [noise=true , address=145.79.227.22:4509 , piece_id=BDZIDW725LYFQF4CZKMOGMTKMSREVVMUQBTIQH5IC43EOHGFJD4A, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:00.529552096 (182h5m0.400635131s) [noise=true , address=145.79.227.42:4123 , piece_id=XMJDDBGGY44FRQF73PSULH5L37MBCNWBJXT62PHX7IKSVJ44S24Q, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:00.542433055 (182h5m0.41351609s) [noise=true , address=77.90.31.10:4520 , piece_id=PTDKHO4QVVWVWCM3TPVNPVSWT7I267WHYI6PZVRILOPH423K2NZQ, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:00.552483503 (182h5m0.423566548s) [noise=true , address=145.79.227.32:4836 , piece_id=DBZUHMNUAVAC6UZUDUOTK4WBLV5C3CNIPC25FJBNWPQOTV2IAX4A, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:00.612273003 (182h5m0.483356048s) [noise=true , address=77.90.31.14:4663 , piece_id=HYNK74E4LI2IRX7YV6YDC5IPINRHXKCCW6GQXZT4ZORGSGM6DURA, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:00.835989661 (182h5m0.707072706s) [noise=true , address=145.79.227.50:4373 , piece_id=HDJ3F7PSHTMXGMKKCZFQ2EVCWDBZGJ5W3I2EOMN46ZBFJNRLVJDQ, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:01.196293326 (182h5m1.067376371s) [noise=true , address=145.79.227.50:4374 , piece_id=2TGRU7HF5SROJ4EEHG32HI2EKAHH4GQL4XXYWMNGSATCYLCNVUFQ, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:01.226460089 (182h5m1.097543135s) [noise=true , address=145.79.227.36:4519 , piece_id=BHHN6RRNKPY57DNH44ILZVXAWL66IHR2A5YQ4DSBE22LQFPZSBDQ, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:01.265467076 (182h5m1.136550121s) [noise=true , address=145.79.227.48:4303 , piece_id=XWN2P6JSSOLG6XTQLNQVF3RHHXJFRIQKDLYDXWOV7UFPV5SK32OQ, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:01.509373308 (182h5m1.380456353s) [noise=true , address=145.79.227.46:4603 , piece_id=WTP7JT4BFDJVR7JHFLC5OIPPJ3CZSCHRCHQDP67QODHH3BJ2GB4Q, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:02.844300840 (182h5m2.715383885s) [noise=true , address=77.90.31.12:4589 , piece_id=TEEE5AM3GVE6QA5XDCT6NEQUSFJGSWRIW4DBFYNXFUEQRXQFI5UA, seg_pos={0 0}, upload=159506]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 13 05:34:24.040889140 (182h5m23.911972185s) [noise=true , address=77.90.31.11:4555 , piece_id=DDDWRVUGDOM6ALDY5K2MWEO2HR62GHFJF5BIWDTUALHHXZRXB4CQ, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:24.040917073 (182h5m23.912000118s) [seg_pos={0 0}, upload=159506]: Segment reached optimal threshold of 13 pieces.
Dec 13 05:34:24.041085103 (182h5m23.912168148s) [seg_pos={0 0}, upload=159506]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 13 05:34:24.041168802 (182h5m23.912251847s) [upload=159506]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 13 05:34:24.041177479 (182h5m23.912260524s) [upload=159506]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 13 05:34:24.348509366 (182h5m24.219592411s) [upload=159506]: Done uploading segments...
Dec 13 05:34:24.348533872 (182h5m24.219616907s) [upload=159506]: Upload finished. err: <nil>
Duration: 8.690449272s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.003426669s trace:
https://jaeger.dev.storj.io/trace/20a7eed07221a46a
FAIL (2025-12-13 05:34:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 13 05:33:58.098067468 (182h4m57.969150503s) [upload=159506]: Starting upload
Dec 13 05:33:58.098123904 (182h4m57.969206949s) [upload=159506]: Done starting upload
Dec 13 05:33:58.098354594 (182h4m57.969437639s) [upload=159506]: Uploading segments...
Dec 13 05:33:58.098654204 (182h4m57.969737239s) [upload=159506]: Got next segment. Inline: false
Dec 13 05:33:58.098692758 (182h4m57.969775793s) [upload=159506]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 13 05:33:58.098697167 (182h4m57.969780202s) [upload=159506]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 13 05:33:58.362129190 (182h4m58.233212245s) [seg_pos={0 0}, upload=159506]: Begin upload segment...
Dec 13 05:33:58.362185707 (182h4m58.233268742s) [seg_pos={0 0}, upload=159506]: Done begin upload segment.
Dec 13 05:33:58.362216586 (182h4m58.233299631s) [noise=true , address=77.90.31.11:4555 , piece_id=DDDWRVUGDOM6ALDY5K2MWEO2HR62GHFJF5BIWDTUALHHXZRXB4CQ, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.362268716 (182h4m58.233351761s) [noise=true , address=77.90.31.10:4520 , piece_id=PTDKHO4QVVWVWCM3TPVNPVSWT7I267WHYI6PZVRILOPH423K2NZQ, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.362778736 (182h4m58.233861771s) [noise=true , address=77.90.31.14:4663 , piece_id=HYNK74E4LI2IRX7YV6YDC5IPINRHXKCCW6GQXZT4ZORGSGM6DURA, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.363444535 (182h4m58.23452758s) [noise=true , address=145.79.227.50:4373 , piece_id=HDJ3F7PSHTMXGMKKCZFQ2EVCWDBZGJ5W3I2EOMN46ZBFJNRLVJDQ, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.363518375 (182h4m58.23460141s) [noise=true , address=77.90.31.16:4734 , piece_id=4VLPNEXKROFP2QXREKBDMKESFGZUU45CTOCUHJ6EWRLCGNGNVO2A, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.363841891 (182h4m58.234924936s) [noise=true , address=145.79.227.40:4049 , piece_id=PASZUKH76WKE62G3UBUXEH32TPF26CS7HQQDKN5UZ5RICWQFQNFA, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.363858633 (182h4m58.234941669s) [noise=true , address=145.79.227.32:4836 , piece_id=DBZUHMNUAVAC6UZUDUOTK4WBLV5C3CNIPC25FJBNWPQOTV2IAX4A, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.364176749 (182h4m58.235259794s) [noise=true , address=77.90.31.12:4589 , piece_id=TEEE5AM3GVE6QA5XDCT6NEQUSFJGSWRIW4DBFYNXFUEQRXQFI5UA, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.364264025 (182h4m58.23534707s) [noise=true , address=145.79.227.50:4374 , piece_id=2TGRU7HF5SROJ4EEHG32HI2EKAHH4GQL4XXYWMNGSATCYLCNVUFQ, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.364451392 (182h4m58.235534427s) [noise=true , address=145.79.227.46:4603 , piece_id=WTP7JT4BFDJVR7JHFLC5OIPPJ3CZSCHRCHQDP67QODHH3BJ2GB4Q, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.364739200 (182h4m58.235822235s) [noise=true , address=145.79.227.36:4519 , piece_id=BHHN6RRNKPY57DNH44ILZVXAWL66IHR2A5YQ4DSBE22LQFPZSBDQ, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.365064920 (182h4m58.236147965s) [noise=true , address=145.79.227.22:4509 , piece_id=BDZIDW725LYFQF4CZKMOGMTKMSREVVMUQBTIQH5IC43EOHGFJD4A, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.365389158 (182h4m58.236472203s) [noise=true , address=145.79.227.48:4303 , piece_id=XWN2P6JSSOLG6XTQLNQVF3RHHXJFRIQKDLYDXWOV7UFPV5SK32OQ, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.365680051 (182h4m58.236763087s) [noise=true , address=145.79.227.48:4302 , piece_id=CDQGUACHANDREDEST3QCZ44CDWL3QARYT4B6EF4QGD4SOPQHETHA, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.365963611 (182h4m58.237046646s) [noise=true , address=145.79.227.42:4123 , piece_id=XMJDDBGGY44FRQF73PSULH5L37MBCNWBJXT62PHX7IKSVJ44S24Q, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:58.366485655 (182h4m58.23756869s) [noise=true , address=145.79.227.30:4763 , piece_id=CAEOGXNNUYUKEVKQMAMI4NTDPTIG67H4TELFXIKJE3SFNGQ4NXJQ, seg_pos={0 0}, upload=159506]: Uploading piece...
Dec 13 05:33:59.473265431 (182h4m59.344348466s) [noise=true , address=77.90.31.16:4734 , piece_id=4VLPNEXKROFP2QXREKBDMKESFGZUU45CTOCUHJ6EWRLCGNGNVO2A, seg_pos={0 0}, upload=159506]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000
Dec 13 05:33:59.786446836 (182h4m59.657529891s) [noise=true , address=145.79.227.30:4763 , piece_id=CAEOGXNNUYUKEVKQMAMI4NTDPTIG67H4TELFXIKJE3SFNGQ4NXJQ, seg_pos={0 0}, upload=159506]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000
Dec 13 05:34:00.203382529 (182h5m0.074465564s) [upload=159506]: Next returned nil segment
Dec 13 05:34:00.203398039 (182h5m0.074481084s) [upload=159506]: Waiting for error group managing segments...
Dec 13 05:34:00.521187459 (182h5m0.392270494s) [noise=true , address=145.79.227.40:4049 , piece_id=PASZUKH76WKE62G3UBUXEH32TPF26CS7HQQDKN5UZ5RICWQFQNFA, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:00.521787982 (182h5m0.392871017s) [noise=true , address=145.79.227.48:4302 , piece_id=CDQGUACHANDREDEST3QCZ44CDWL3QARYT4B6EF4QGD4SOPQHETHA, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:00.525174060 (182h5m0.396257105s) [noise=true , address=145.79.227.22:4509 , piece_id=BDZIDW725LYFQF4CZKMOGMTKMSREVVMUQBTIQH5IC43EOHGFJD4A, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:00.529552096 (182h5m0.400635131s) [noise=true , address=145.79.227.42:4123 , piece_id=XMJDDBGGY44FRQF73PSULH5L37MBCNWBJXT62PHX7IKSVJ44S24Q, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:00.542433055 (182h5m0.41351609s) [noise=true , address=77.90.31.10:4520 , piece_id=PTDKHO4QVVWVWCM3TPVNPVSWT7I267WHYI6PZVRILOPH423K2NZQ, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:00.552483503 (182h5m0.423566548s) [noise=true , address=145.79.227.32:4836 , piece_id=DBZUHMNUAVAC6UZUDUOTK4WBLV5C3CNIPC25FJBNWPQOTV2IAX4A, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:00.612273003 (182h5m0.483356048s) [noise=true , address=77.90.31.14:4663 , piece_id=HYNK74E4LI2IRX7YV6YDC5IPINRHXKCCW6GQXZT4ZORGSGM6DURA, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:00.835989661 (182h5m0.707072706s) [noise=true , address=145.79.227.50:4373 , piece_id=HDJ3F7PSHTMXGMKKCZFQ2EVCWDBZGJ5W3I2EOMN46ZBFJNRLVJDQ, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:01.196293326 (182h5m1.067376371s) [noise=true , address=145.79.227.50:4374 , piece_id=2TGRU7HF5SROJ4EEHG32HI2EKAHH4GQL4XXYWMNGSATCYLCNVUFQ, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:01.226460089 (182h5m1.097543135s) [noise=true , address=145.79.227.36:4519 , piece_id=BHHN6RRNKPY57DNH44ILZVXAWL66IHR2A5YQ4DSBE22LQFPZSBDQ, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:01.265467076 (182h5m1.136550121s) [noise=true , address=145.79.227.48:4303 , piece_id=XWN2P6JSSOLG6XTQLNQVF3RHHXJFRIQKDLYDXWOV7UFPV5SK32OQ, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:01.509373308 (182h5m1.380456353s) [noise=true , address=145.79.227.46:4603 , piece_id=WTP7JT4BFDJVR7JHFLC5OIPPJ3CZSCHRCHQDP67QODHH3BJ2GB4Q, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:02.844300840 (182h5m2.715383885s) [noise=true , address=77.90.31.12:4589 , piece_id=TEEE5AM3GVE6QA5XDCT6NEQUSFJGSWRIW4DBFYNXFUEQRXQFI5UA, seg_pos={0 0}, upload=159506]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 13 05:34:24.040889140 (182h5m23.911972185s) [noise=true , address=77.90.31.11:4555 , piece_id=DDDWRVUGDOM6ALDY5K2MWEO2HR62GHFJF5BIWDTUALHHXZRXB4CQ, seg_pos={0 0}, upload=159506]: Done uploading piece. err: <nil>
Dec 13 05:34:24.040917073 (182h5m23.912000118s) [seg_pos={0 0}, upload=159506]: Segment reached optimal threshold of 13 pieces.
Dec 13 05:34:24.041085103 (182h5m23.912168148s) [seg_pos={0 0}, upload=159506]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 13 05:34:24.041168802 (182h5m23.912251847s) [upload=159506]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 13 05:34:24.041177479 (182h5m23.912260524s) [upload=159506]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 13 05:34:24.348509366 (182h5m24.219592411s) [upload=159506]: Done uploading segments...
Dec 13 05:34:24.348533872 (182h5m24.219616907s) [upload=159506]: Upload finished. err: <nil>
Duration: 30.003449112s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
6.066895193s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:59:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.066920592s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
5.931525445s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:59:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.931566084s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
6.142324419s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:58:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.142351741s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
6.135860506s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:58:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.135883491s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
6.109345547s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:57:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.109368072s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
6.50756037s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:56:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.507582071s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
6.100132919s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:55:53 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.100189617s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
6.443274837s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:54:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.44329703s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
6.221584734s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:53:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.222692793s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
6.033662344s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:53:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.03369118s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
6.737193905s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:52:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.737215125s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
5.725926869s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:51:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.725956766s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
6.155284513s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:50:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.155312306s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
6.213979432s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:49:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.214009808s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
6.230758007s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:49:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.230788695s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
6.772405874s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:48:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.772452684s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
6.064441788s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:47:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.06446367s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
6.224702106s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:46:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.224749306s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
6.426039892s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:46:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.426109044s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
5.728319169s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:44:53 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.728353335s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
6.279164741s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:44:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.279260815s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
5.846654731s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:43:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.846686581s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
6.658383175s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:42:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.658483236s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
7.285581757s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:41:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.285605512s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
6.790046178s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:40:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.790074391s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
6.389064692s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:39:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.389146117s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
6.190932202s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:39:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.190957871s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
6.838490393s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:38:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.838518848s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
5.816069388s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:38:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.81609173s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
5.997535583s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:37:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.997582532s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
6.643723559s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:36:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.64391269s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
5.933045982s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:35:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.933076469s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
5.764703033s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:34:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.764719183s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
6.126701434s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:34:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.126731972s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
6.409101962s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:33:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.409128854s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
5.833873002s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:32:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.833901204s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
7.116872495s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:31:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.116903734s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
7.696627922s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:30:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.69667367s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
5.87492071s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:29:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.874944917s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
5.854702293s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:29:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.854715537s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
6.34410347s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:29:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.344130119s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
6.30136193s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:28:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.30139364s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
7.966262939s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:27:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.966284489s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
6.022287303s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:26:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.02231656s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
6.206195079s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:25:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.206221358s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
5.690829314s trace:
https://jaeger.dev.storj.io/trace/615b33e8e064003c
FAIL (2025-12-13 04:25:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.690963561s
edge
gateway
us-select-1
|