Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 9.611834386s trace: https://jaeger.dev.storj.io/trace/397b46dd018f3fc5 FAIL (2025-12-10 07:11:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 06:30:35.391807620 (111h1m35.262890665s) [upload=96845]: Starting upload Dec 10 06:30:35.391896590 (111h1m35.262979655s) [upload=96845]: Done starting upload Dec 10 06:30:35.391969459 (111h1m35.263052504s) [upload=96845]: Uploading segments... Dec 10 06:30:35.392189627 (111h1m35.263272663s) [upload=96845]: Got next segment. Inline: false Dec 10 06:30:35.392217962 (111h1m35.263300997s) [upload=96845]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 06:30:35.392223091 (111h1m35.263306126s) [upload=96845]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 06:30:35.639824280 (111h1m35.510907325s) [seg_pos={0 0}, upload=96845]: Begin upload segment... Dec 10 06:30:35.639915774 (111h1m35.51099882s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.639970037 (111h1m35.511053072s) [seg_pos={0 0}, upload=96845]: Done begin upload segment. Dec 10 06:30:35.640008270 (111h1m35.511091305s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640528070 (111h1m35.511611105s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640677645 (111h1m35.51176068s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641070303 (111h1m35.512153348s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641398267 (111h1m35.512481312s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641982800 (111h1m35.513065845s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642065027 (111h1m35.513148062s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642657175 (111h1m35.51374021s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642738600 (111h1m35.513821645s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.643375131 (111h1m35.514458166s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644145648 (111h1m35.515228683s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644951062 (111h1m35.516034107s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.647301418 (111h1m35.518384453s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.648638635 (111h1m35.51972168s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.650012029 (111h1m35.521095075s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:36.662681817 (111h1m36.533764862s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.773459999 (111h1m36.644543034s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.788144411 (111h1m36.659227466s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:37.344090880 (111h1m37.215173935s) [upload=96845]: Next returned nil segment Dec 10 06:30:37.344104926 (111h1m37.215187971s) [upload=96845]: Waiting for error group managing segments... Dec 10 06:30:37.629121993 (111h1m37.500205028s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.631506683 (111h1m37.502589718s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.639977544 (111h1m37.511060589s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.646143775 (111h1m37.51722681s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.657520980 (111h1m37.528604015s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.659491392 (111h1m37.530574427s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.663175928 (111h1m37.534258963s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.665314009 (111h1m37.536397054s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.670795998 (111h1m37.541879033s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.679195422 (111h1m37.550278467s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.828334161 (111h1m37.699417196s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.834834629 (111h1m37.705917664s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:31:04.695992065 (111h2m4.56707511s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 10 06:31:04.696053741 (111h2m4.567136776s) [seg_pos={0 0}, upload=96845]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 06:31:04.939762863 (111h2m4.810845908s) [upload=96845]: Done uploading segments... Dec 10 06:31:04.939817447 (111h2m4.810900492s) [upload=96845]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.61187841s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 15.612750766s trace: https://jaeger.dev.storj.io/trace/397b46dd018f3fc5 FAIL (2025-12-10 07:11:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 06:30:35.391807620 (111h1m35.262890665s) [upload=96845]: Starting upload Dec 10 06:30:35.391896590 (111h1m35.262979655s) [upload=96845]: Done starting upload Dec 10 06:30:35.391969459 (111h1m35.263052504s) [upload=96845]: Uploading segments... Dec 10 06:30:35.392189627 (111h1m35.263272663s) [upload=96845]: Got next segment. Inline: false Dec 10 06:30:35.392217962 (111h1m35.263300997s) [upload=96845]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 06:30:35.392223091 (111h1m35.263306126s) [upload=96845]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 06:30:35.639824280 (111h1m35.510907325s) [seg_pos={0 0}, upload=96845]: Begin upload segment... Dec 10 06:30:35.639915774 (111h1m35.51099882s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.639970037 (111h1m35.511053072s) [seg_pos={0 0}, upload=96845]: Done begin upload segment. Dec 10 06:30:35.640008270 (111h1m35.511091305s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640528070 (111h1m35.511611105s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640677645 (111h1m35.51176068s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641070303 (111h1m35.512153348s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641398267 (111h1m35.512481312s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641982800 (111h1m35.513065845s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642065027 (111h1m35.513148062s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642657175 (111h1m35.51374021s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642738600 (111h1m35.513821645s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.643375131 (111h1m35.514458166s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644145648 (111h1m35.515228683s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644951062 (111h1m35.516034107s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.647301418 (111h1m35.518384453s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.648638635 (111h1m35.51972168s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.650012029 (111h1m35.521095075s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:36.662681817 (111h1m36.533764862s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.773459999 (111h1m36.644543034s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.788144411 (111h1m36.659227466s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:37.344090880 (111h1m37.215173935s) [upload=96845]: Next returned nil segment Dec 10 06:30:37.344104926 (111h1m37.215187971s) [upload=96845]: Waiting for error group managing segments... Dec 10 06:30:37.629121993 (111h1m37.500205028s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.631506683 (111h1m37.502589718s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.639977544 (111h1m37.511060589s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.646143775 (111h1m37.51722681s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.657520980 (111h1m37.528604015s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.659491392 (111h1m37.530574427s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.663175928 (111h1m37.534258963s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.665314009 (111h1m37.536397054s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.670795998 (111h1m37.541879033s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.679195422 (111h1m37.550278467s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.828334161 (111h1m37.699417196s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.834834629 (111h1m37.705917664s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:31:04.695992065 (111h2m4.56707511s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 10 06:31:04.696053741 (111h2m4.567136776s) [seg_pos={0 0}, upload=96845]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 06:31:04.939762863 (111h2m4.810845908s) [upload=96845]: Done uploading segments... Dec 10 06:31:04.939817447 (111h2m4.810900492s) [upload=96845]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 15.612778378s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.................X...X......X...XX...X.XXXX...X.......X.....] duration: 4.047123181s trace: https://jaeger.dev.storj.io/trace/cb3bf95f96d5085 FAIL (2025-12-10 07:10:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.047152496s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 9.411999012s trace: https://jaeger.dev.storj.io/trace/397b46dd018f3fc5 FAIL (2025-12-10 07:10:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 06:30:35.391807620 (111h1m35.262890665s) [upload=96845]: Starting upload Dec 10 06:30:35.391896590 (111h1m35.262979655s) [upload=96845]: Done starting upload Dec 10 06:30:35.391969459 (111h1m35.263052504s) [upload=96845]: Uploading segments... Dec 10 06:30:35.392189627 (111h1m35.263272663s) [upload=96845]: Got next segment. Inline: false Dec 10 06:30:35.392217962 (111h1m35.263300997s) [upload=96845]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 06:30:35.392223091 (111h1m35.263306126s) [upload=96845]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 06:30:35.639824280 (111h1m35.510907325s) [seg_pos={0 0}, upload=96845]: Begin upload segment... Dec 10 06:30:35.639915774 (111h1m35.51099882s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.639970037 (111h1m35.511053072s) [seg_pos={0 0}, upload=96845]: Done begin upload segment. Dec 10 06:30:35.640008270 (111h1m35.511091305s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640528070 (111h1m35.511611105s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640677645 (111h1m35.51176068s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641070303 (111h1m35.512153348s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641398267 (111h1m35.512481312s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641982800 (111h1m35.513065845s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642065027 (111h1m35.513148062s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642657175 (111h1m35.51374021s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642738600 (111h1m35.513821645s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.643375131 (111h1m35.514458166s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644145648 (111h1m35.515228683s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644951062 (111h1m35.516034107s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.647301418 (111h1m35.518384453s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.648638635 (111h1m35.51972168s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.650012029 (111h1m35.521095075s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:36.662681817 (111h1m36.533764862s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.773459999 (111h1m36.644543034s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.788144411 (111h1m36.659227466s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:37.344090880 (111h1m37.215173935s) [upload=96845]: Next returned nil segment Dec 10 06:30:37.344104926 (111h1m37.215187971s) [upload=96845]: Waiting for error group managing segments... Dec 10 06:30:37.629121993 (111h1m37.500205028s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.631506683 (111h1m37.502589718s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.639977544 (111h1m37.511060589s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.646143775 (111h1m37.51722681s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.657520980 (111h1m37.528604015s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.659491392 (111h1m37.530574427s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.663175928 (111h1m37.534258963s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.665314009 (111h1m37.536397054s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.670795998 (111h1m37.541879033s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.679195422 (111h1m37.550278467s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.828334161 (111h1m37.699417196s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.834834629 (111h1m37.705917664s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:31:04.695992065 (111h2m4.56707511s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 10 06:31:04.696053741 (111h2m4.567136776s) [seg_pos={0 0}, upload=96845]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 06:31:04.939762863 (111h2m4.810845908s) [upload=96845]: Done uploading segments... Dec 10 06:31:04.939817447 (111h2m4.810900492s) [upload=96845]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.412027104s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [..................X...X......X...XX...X.XXXX...X.......X....] duration: 4.075686781s trace: https://jaeger.dev.storj.io/trace/cb3bf95f96d5085 FAIL (2025-12-10 07:10:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.075718391s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 8.723661995s trace: https://jaeger.dev.storj.io/trace/397b46dd018f3fc5 FAIL (2025-12-10 07:09:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 06:30:35.391807620 (111h1m35.262890665s) [upload=96845]: Starting upload Dec 10 06:30:35.391896590 (111h1m35.262979655s) [upload=96845]: Done starting upload Dec 10 06:30:35.391969459 (111h1m35.263052504s) [upload=96845]: Uploading segments... Dec 10 06:30:35.392189627 (111h1m35.263272663s) [upload=96845]: Got next segment. Inline: false Dec 10 06:30:35.392217962 (111h1m35.263300997s) [upload=96845]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 06:30:35.392223091 (111h1m35.263306126s) [upload=96845]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 06:30:35.639824280 (111h1m35.510907325s) [seg_pos={0 0}, upload=96845]: Begin upload segment... Dec 10 06:30:35.639915774 (111h1m35.51099882s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.639970037 (111h1m35.511053072s) [seg_pos={0 0}, upload=96845]: Done begin upload segment. Dec 10 06:30:35.640008270 (111h1m35.511091305s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640528070 (111h1m35.511611105s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640677645 (111h1m35.51176068s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641070303 (111h1m35.512153348s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641398267 (111h1m35.512481312s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641982800 (111h1m35.513065845s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642065027 (111h1m35.513148062s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642657175 (111h1m35.51374021s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642738600 (111h1m35.513821645s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.643375131 (111h1m35.514458166s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644145648 (111h1m35.515228683s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644951062 (111h1m35.516034107s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.647301418 (111h1m35.518384453s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.648638635 (111h1m35.51972168s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.650012029 (111h1m35.521095075s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:36.662681817 (111h1m36.533764862s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.773459999 (111h1m36.644543034s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.788144411 (111h1m36.659227466s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:37.344090880 (111h1m37.215173935s) [upload=96845]: Next returned nil segment Dec 10 06:30:37.344104926 (111h1m37.215187971s) [upload=96845]: Waiting for error group managing segments... Dec 10 06:30:37.629121993 (111h1m37.500205028s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.631506683 (111h1m37.502589718s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.639977544 (111h1m37.511060589s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.646143775 (111h1m37.51722681s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.657520980 (111h1m37.528604015s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.659491392 (111h1m37.530574427s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.663175928 (111h1m37.534258963s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.665314009 (111h1m37.536397054s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.670795998 (111h1m37.541879033s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.679195422 (111h1m37.550278467s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.828334161 (111h1m37.699417196s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.834834629 (111h1m37.705917664s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:31:04.695992065 (111h2m4.56707511s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 10 06:31:04.696053741 (111h2m4.567136776s) [seg_pos={0 0}, upload=96845]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 06:31:04.939762863 (111h2m4.810845908s) [upload=96845]: Done uploading segments... Dec 10 06:31:04.939817447 (111h2m4.810900492s) [upload=96845]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.723718725s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 11.408403565s trace: https://jaeger.dev.storj.io/trace/397b46dd018f3fc5 FAIL (2025-12-10 07:09:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 06:30:35.391807620 (111h1m35.262890665s) [upload=96845]: Starting upload Dec 10 06:30:35.391896590 (111h1m35.262979655s) [upload=96845]: Done starting upload Dec 10 06:30:35.391969459 (111h1m35.263052504s) [upload=96845]: Uploading segments... Dec 10 06:30:35.392189627 (111h1m35.263272663s) [upload=96845]: Got next segment. Inline: false Dec 10 06:30:35.392217962 (111h1m35.263300997s) [upload=96845]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 06:30:35.392223091 (111h1m35.263306126s) [upload=96845]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 06:30:35.639824280 (111h1m35.510907325s) [seg_pos={0 0}, upload=96845]: Begin upload segment... Dec 10 06:30:35.639915774 (111h1m35.51099882s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.639970037 (111h1m35.511053072s) [seg_pos={0 0}, upload=96845]: Done begin upload segment. Dec 10 06:30:35.640008270 (111h1m35.511091305s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640528070 (111h1m35.511611105s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640677645 (111h1m35.51176068s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641070303 (111h1m35.512153348s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641398267 (111h1m35.512481312s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641982800 (111h1m35.513065845s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642065027 (111h1m35.513148062s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642657175 (111h1m35.51374021s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642738600 (111h1m35.513821645s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.643375131 (111h1m35.514458166s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644145648 (111h1m35.515228683s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644951062 (111h1m35.516034107s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.647301418 (111h1m35.518384453s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.648638635 (111h1m35.51972168s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.650012029 (111h1m35.521095075s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:36.662681817 (111h1m36.533764862s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.773459999 (111h1m36.644543034s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.788144411 (111h1m36.659227466s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:37.344090880 (111h1m37.215173935s) [upload=96845]: Next returned nil segment Dec 10 06:30:37.344104926 (111h1m37.215187971s) [upload=96845]: Waiting for error group managing segments... Dec 10 06:30:37.629121993 (111h1m37.500205028s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.631506683 (111h1m37.502589718s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.639977544 (111h1m37.511060589s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.646143775 (111h1m37.51722681s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.657520980 (111h1m37.528604015s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.659491392 (111h1m37.530574427s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.663175928 (111h1m37.534258963s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.665314009 (111h1m37.536397054s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.670795998 (111h1m37.541879033s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.679195422 (111h1m37.550278467s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.828334161 (111h1m37.699417196s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.834834629 (111h1m37.705917664s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:31:04.695992065 (111h2m4.56707511s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 10 06:31:04.696053741 (111h2m4.567136776s) [seg_pos={0 0}, upload=96845]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 06:31:04.939762863 (111h2m4.810845908s) [upload=96845]: Done uploading segments... Dec 10 06:31:04.939817447 (111h2m4.810900492s) [upload=96845]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 11.408428201s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [...................X...X......X...XX...X.XXXX...X.......X...] duration: 3.940783438s trace: https://jaeger.dev.storj.io/trace/cb3bf95f96d5085 FAIL (2025-12-10 07:09:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.940813074s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [....................X...X......X...XX...X.XXXX...X.......X..] duration: 4.025641736s trace: https://jaeger.dev.storj.io/trace/cb3bf95f96d5085 FAIL (2025-12-10 07:08:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.025672995s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 10.48392959s trace: https://jaeger.dev.storj.io/trace/397b46dd018f3fc5 FAIL (2025-12-10 07:08:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 06:30:35.391807620 (111h1m35.262890665s) [upload=96845]: Starting upload Dec 10 06:30:35.391896590 (111h1m35.262979655s) [upload=96845]: Done starting upload Dec 10 06:30:35.391969459 (111h1m35.263052504s) [upload=96845]: Uploading segments... Dec 10 06:30:35.392189627 (111h1m35.263272663s) [upload=96845]: Got next segment. Inline: false Dec 10 06:30:35.392217962 (111h1m35.263300997s) [upload=96845]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 06:30:35.392223091 (111h1m35.263306126s) [upload=96845]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 06:30:35.639824280 (111h1m35.510907325s) [seg_pos={0 0}, upload=96845]: Begin upload segment... Dec 10 06:30:35.639915774 (111h1m35.51099882s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.639970037 (111h1m35.511053072s) [seg_pos={0 0}, upload=96845]: Done begin upload segment. Dec 10 06:30:35.640008270 (111h1m35.511091305s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640528070 (111h1m35.511611105s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640677645 (111h1m35.51176068s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641070303 (111h1m35.512153348s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641398267 (111h1m35.512481312s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641982800 (111h1m35.513065845s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642065027 (111h1m35.513148062s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642657175 (111h1m35.51374021s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642738600 (111h1m35.513821645s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.643375131 (111h1m35.514458166s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644145648 (111h1m35.515228683s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644951062 (111h1m35.516034107s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.647301418 (111h1m35.518384453s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.648638635 (111h1m35.51972168s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.650012029 (111h1m35.521095075s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:36.662681817 (111h1m36.533764862s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.773459999 (111h1m36.644543034s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.788144411 (111h1m36.659227466s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:37.344090880 (111h1m37.215173935s) [upload=96845]: Next returned nil segment Dec 10 06:30:37.344104926 (111h1m37.215187971s) [upload=96845]: Waiting for error group managing segments... Dec 10 06:30:37.629121993 (111h1m37.500205028s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.631506683 (111h1m37.502589718s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.639977544 (111h1m37.511060589s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.646143775 (111h1m37.51722681s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.657520980 (111h1m37.528604015s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.659491392 (111h1m37.530574427s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.663175928 (111h1m37.534258963s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.665314009 (111h1m37.536397054s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.670795998 (111h1m37.541879033s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.679195422 (111h1m37.550278467s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.828334161 (111h1m37.699417196s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.834834629 (111h1m37.705917664s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:31:04.695992065 (111h2m4.56707511s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 10 06:31:04.696053741 (111h2m4.567136776s) [seg_pos={0 0}, upload=96845]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 06:31:04.939762863 (111h2m4.810845908s) [upload=96845]: Done uploading segments... Dec 10 06:31:04.939817447 (111h2m4.810900492s) [upload=96845]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 10.483961641s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.....................X...X......X...XX...X.XXXX...X.......X.] duration: 4.896111526s trace: https://jaeger.dev.storj.io/trace/cb3bf95f96d5085 FAIL (2025-12-10 07:08:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.896142193s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 9.408727722s trace: https://jaeger.dev.storj.io/trace/397b46dd018f3fc5 FAIL (2025-12-10 07:07:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 06:30:35.391807620 (111h1m35.262890665s) [upload=96845]: Starting upload Dec 10 06:30:35.391896590 (111h1m35.262979655s) [upload=96845]: Done starting upload Dec 10 06:30:35.391969459 (111h1m35.263052504s) [upload=96845]: Uploading segments... Dec 10 06:30:35.392189627 (111h1m35.263272663s) [upload=96845]: Got next segment. Inline: false Dec 10 06:30:35.392217962 (111h1m35.263300997s) [upload=96845]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 06:30:35.392223091 (111h1m35.263306126s) [upload=96845]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 06:30:35.639824280 (111h1m35.510907325s) [seg_pos={0 0}, upload=96845]: Begin upload segment... Dec 10 06:30:35.639915774 (111h1m35.51099882s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.639970037 (111h1m35.511053072s) [seg_pos={0 0}, upload=96845]: Done begin upload segment. Dec 10 06:30:35.640008270 (111h1m35.511091305s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640528070 (111h1m35.511611105s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640677645 (111h1m35.51176068s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641070303 (111h1m35.512153348s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641398267 (111h1m35.512481312s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641982800 (111h1m35.513065845s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642065027 (111h1m35.513148062s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642657175 (111h1m35.51374021s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642738600 (111h1m35.513821645s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.643375131 (111h1m35.514458166s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644145648 (111h1m35.515228683s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644951062 (111h1m35.516034107s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.647301418 (111h1m35.518384453s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.648638635 (111h1m35.51972168s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.650012029 (111h1m35.521095075s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:36.662681817 (111h1m36.533764862s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.773459999 (111h1m36.644543034s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.788144411 (111h1m36.659227466s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:37.344090880 (111h1m37.215173935s) [upload=96845]: Next returned nil segment Dec 10 06:30:37.344104926 (111h1m37.215187971s) [upload=96845]: Waiting for error group managing segments... Dec 10 06:30:37.629121993 (111h1m37.500205028s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.631506683 (111h1m37.502589718s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.639977544 (111h1m37.511060589s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.646143775 (111h1m37.51722681s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.657520980 (111h1m37.528604015s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.659491392 (111h1m37.530574427s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.663175928 (111h1m37.534258963s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.665314009 (111h1m37.536397054s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.670795998 (111h1m37.541879033s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.679195422 (111h1m37.550278467s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.828334161 (111h1m37.699417196s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.834834629 (111h1m37.705917664s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:31:04.695992065 (111h2m4.56707511s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 10 06:31:04.696053741 (111h2m4.567136776s) [seg_pos={0 0}, upload=96845]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 06:31:04.939762863 (111h2m4.810845908s) [upload=96845]: Done uploading segments... Dec 10 06:31:04.939817447 (111h2m4.810900492s) [upload=96845]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.408756637s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [......................X...X......X...XX...X.XXXX...X.......X] duration: 5.000871129s trace: https://jaeger.dev.storj.io/trace/cb3bf95f96d5085 FAIL (2025-12-10 07:07:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001118358s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 9.477992812s trace: https://jaeger.dev.storj.io/trace/397b46dd018f3fc5 FAIL (2025-12-10 07:06:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 06:30:35.391807620 (111h1m35.262890665s) [upload=96845]: Starting upload Dec 10 06:30:35.391896590 (111h1m35.262979655s) [upload=96845]: Done starting upload Dec 10 06:30:35.391969459 (111h1m35.263052504s) [upload=96845]: Uploading segments... Dec 10 06:30:35.392189627 (111h1m35.263272663s) [upload=96845]: Got next segment. Inline: false Dec 10 06:30:35.392217962 (111h1m35.263300997s) [upload=96845]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 06:30:35.392223091 (111h1m35.263306126s) [upload=96845]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 06:30:35.639824280 (111h1m35.510907325s) [seg_pos={0 0}, upload=96845]: Begin upload segment... Dec 10 06:30:35.639915774 (111h1m35.51099882s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.639970037 (111h1m35.511053072s) [seg_pos={0 0}, upload=96845]: Done begin upload segment. Dec 10 06:30:35.640008270 (111h1m35.511091305s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640528070 (111h1m35.511611105s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640677645 (111h1m35.51176068s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641070303 (111h1m35.512153348s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641398267 (111h1m35.512481312s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641982800 (111h1m35.513065845s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642065027 (111h1m35.513148062s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642657175 (111h1m35.51374021s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642738600 (111h1m35.513821645s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.643375131 (111h1m35.514458166s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644145648 (111h1m35.515228683s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644951062 (111h1m35.516034107s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.647301418 (111h1m35.518384453s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.648638635 (111h1m35.51972168s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.650012029 (111h1m35.521095075s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:36.662681817 (111h1m36.533764862s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.773459999 (111h1m36.644543034s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.788144411 (111h1m36.659227466s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:37.344090880 (111h1m37.215173935s) [upload=96845]: Next returned nil segment Dec 10 06:30:37.344104926 (111h1m37.215187971s) [upload=96845]: Waiting for error group managing segments... Dec 10 06:30:37.629121993 (111h1m37.500205028s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.631506683 (111h1m37.502589718s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.639977544 (111h1m37.511060589s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.646143775 (111h1m37.51722681s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.657520980 (111h1m37.528604015s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.659491392 (111h1m37.530574427s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.663175928 (111h1m37.534258963s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.665314009 (111h1m37.536397054s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.670795998 (111h1m37.541879033s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.679195422 (111h1m37.550278467s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.828334161 (111h1m37.699417196s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.834834629 (111h1m37.705917664s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:31:04.695992065 (111h2m4.56707511s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 10 06:31:04.696053741 (111h2m4.567136776s) [seg_pos={0 0}, upload=96845]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 06:31:04.939762863 (111h2m4.810845908s) [upload=96845]: Done uploading segments... Dec 10 06:31:04.939817447 (111h2m4.810900492s) [upload=96845]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.478012729s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 9.203987638s trace: https://jaeger.dev.storj.io/trace/397b46dd018f3fc5 FAIL (2025-12-10 07:06:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 06:30:35.391807620 (111h1m35.262890665s) [upload=96845]: Starting upload Dec 10 06:30:35.391896590 (111h1m35.262979655s) [upload=96845]: Done starting upload Dec 10 06:30:35.391969459 (111h1m35.263052504s) [upload=96845]: Uploading segments... Dec 10 06:30:35.392189627 (111h1m35.263272663s) [upload=96845]: Got next segment. Inline: false Dec 10 06:30:35.392217962 (111h1m35.263300997s) [upload=96845]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 06:30:35.392223091 (111h1m35.263306126s) [upload=96845]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 06:30:35.639824280 (111h1m35.510907325s) [seg_pos={0 0}, upload=96845]: Begin upload segment... Dec 10 06:30:35.639915774 (111h1m35.51099882s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.639970037 (111h1m35.511053072s) [seg_pos={0 0}, upload=96845]: Done begin upload segment. Dec 10 06:30:35.640008270 (111h1m35.511091305s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640528070 (111h1m35.511611105s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640677645 (111h1m35.51176068s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641070303 (111h1m35.512153348s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641398267 (111h1m35.512481312s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641982800 (111h1m35.513065845s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642065027 (111h1m35.513148062s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642657175 (111h1m35.51374021s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642738600 (111h1m35.513821645s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.643375131 (111h1m35.514458166s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644145648 (111h1m35.515228683s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644951062 (111h1m35.516034107s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.647301418 (111h1m35.518384453s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.648638635 (111h1m35.51972168s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.650012029 (111h1m35.521095075s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:36.662681817 (111h1m36.533764862s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.773459999 (111h1m36.644543034s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.788144411 (111h1m36.659227466s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:37.344090880 (111h1m37.215173935s) [upload=96845]: Next returned nil segment Dec 10 06:30:37.344104926 (111h1m37.215187971s) [upload=96845]: Waiting for error group managing segments... Dec 10 06:30:37.629121993 (111h1m37.500205028s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.631506683 (111h1m37.502589718s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.639977544 (111h1m37.511060589s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.646143775 (111h1m37.51722681s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.657520980 (111h1m37.528604015s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.659491392 (111h1m37.530574427s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.663175928 (111h1m37.534258963s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.665314009 (111h1m37.536397054s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.670795998 (111h1m37.541879033s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.679195422 (111h1m37.550278467s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.828334161 (111h1m37.699417196s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.834834629 (111h1m37.705917664s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:31:04.695992065 (111h2m4.56707511s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 10 06:31:04.696053741 (111h2m4.567136776s) [seg_pos={0 0}, upload=96845]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 06:31:04.939762863 (111h2m4.810845908s) [upload=96845]: Done uploading segments... Dec 10 06:31:04.939817447 (111h2m4.810900492s) [upload=96845]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.204011153s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.......................X...X......X...XX...X.XXXX...X.......] duration: 4.908368084s trace: https://jaeger.dev.storj.io/trace/1e2823346b2516c6 FAIL (2025-12-10 07:06:13 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.90839221s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [........................X...X......X...XX...X.XXXX...X......] duration: 3.992980623s trace: https://jaeger.dev.storj.io/trace/1e2823346b2516c6 FAIL (2025-12-10 07:05:43 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.993011523s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 7.048476207s trace: https://jaeger.dev.storj.io/trace/397b46dd018f3fc5 FAIL (2025-12-10 07:05:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 06:30:35.391807620 (111h1m35.262890665s) [upload=96845]: Starting upload Dec 10 06:30:35.391896590 (111h1m35.262979655s) [upload=96845]: Done starting upload Dec 10 06:30:35.391969459 (111h1m35.263052504s) [upload=96845]: Uploading segments... Dec 10 06:30:35.392189627 (111h1m35.263272663s) [upload=96845]: Got next segment. Inline: false Dec 10 06:30:35.392217962 (111h1m35.263300997s) [upload=96845]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 06:30:35.392223091 (111h1m35.263306126s) [upload=96845]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 06:30:35.639824280 (111h1m35.510907325s) [seg_pos={0 0}, upload=96845]: Begin upload segment... Dec 10 06:30:35.639915774 (111h1m35.51099882s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.639970037 (111h1m35.511053072s) [seg_pos={0 0}, upload=96845]: Done begin upload segment. Dec 10 06:30:35.640008270 (111h1m35.511091305s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640528070 (111h1m35.511611105s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640677645 (111h1m35.51176068s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641070303 (111h1m35.512153348s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641398267 (111h1m35.512481312s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641982800 (111h1m35.513065845s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642065027 (111h1m35.513148062s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642657175 (111h1m35.51374021s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642738600 (111h1m35.513821645s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.643375131 (111h1m35.514458166s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644145648 (111h1m35.515228683s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644951062 (111h1m35.516034107s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.647301418 (111h1m35.518384453s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.648638635 (111h1m35.51972168s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.650012029 (111h1m35.521095075s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:36.662681817 (111h1m36.533764862s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.773459999 (111h1m36.644543034s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.788144411 (111h1m36.659227466s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:37.344090880 (111h1m37.215173935s) [upload=96845]: Next returned nil segment Dec 10 06:30:37.344104926 (111h1m37.215187971s) [upload=96845]: Waiting for error group managing segments... Dec 10 06:30:37.629121993 (111h1m37.500205028s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.631506683 (111h1m37.502589718s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.639977544 (111h1m37.511060589s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.646143775 (111h1m37.51722681s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.657520980 (111h1m37.528604015s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.659491392 (111h1m37.530574427s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.663175928 (111h1m37.534258963s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.665314009 (111h1m37.536397054s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.670795998 (111h1m37.541879033s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.679195422 (111h1m37.550278467s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.828334161 (111h1m37.699417196s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.834834629 (111h1m37.705917664s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:31:04.695992065 (111h2m4.56707511s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 10 06:31:04.696053741 (111h2m4.567136776s) [seg_pos={0 0}, upload=96845]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 06:31:04.939762863 (111h2m4.810845908s) [upload=96845]: Done uploading segments... Dec 10 06:31:04.939817447 (111h2m4.810900492s) [upload=96845]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.048502928s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.........................X...X......X...XX...X.XXXX...X.....] duration: 4.030008845s trace: https://jaeger.dev.storj.io/trace/1e2823346b2516c6 FAIL (2025-12-10 07:05:04 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.030042759s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.860577879s trace: https://jaeger.dev.storj.io/trace/397b46dd018f3fc5 FAIL (2025-12-10 07:04:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 06:30:35.391807620 (111h1m35.262890665s) [upload=96845]: Starting upload Dec 10 06:30:35.391896590 (111h1m35.262979655s) [upload=96845]: Done starting upload Dec 10 06:30:35.391969459 (111h1m35.263052504s) [upload=96845]: Uploading segments... Dec 10 06:30:35.392189627 (111h1m35.263272663s) [upload=96845]: Got next segment. Inline: false Dec 10 06:30:35.392217962 (111h1m35.263300997s) [upload=96845]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 06:30:35.392223091 (111h1m35.263306126s) [upload=96845]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 06:30:35.639824280 (111h1m35.510907325s) [seg_pos={0 0}, upload=96845]: Begin upload segment... Dec 10 06:30:35.639915774 (111h1m35.51099882s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.639970037 (111h1m35.511053072s) [seg_pos={0 0}, upload=96845]: Done begin upload segment. Dec 10 06:30:35.640008270 (111h1m35.511091305s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640528070 (111h1m35.511611105s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640677645 (111h1m35.51176068s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641070303 (111h1m35.512153348s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641398267 (111h1m35.512481312s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641982800 (111h1m35.513065845s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642065027 (111h1m35.513148062s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642657175 (111h1m35.51374021s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642738600 (111h1m35.513821645s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.643375131 (111h1m35.514458166s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644145648 (111h1m35.515228683s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644951062 (111h1m35.516034107s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.647301418 (111h1m35.518384453s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.648638635 (111h1m35.51972168s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.650012029 (111h1m35.521095075s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:36.662681817 (111h1m36.533764862s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.773459999 (111h1m36.644543034s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.788144411 (111h1m36.659227466s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:37.344090880 (111h1m37.215173935s) [upload=96845]: Next returned nil segment Dec 10 06:30:37.344104926 (111h1m37.215187971s) [upload=96845]: Waiting for error group managing segments... Dec 10 06:30:37.629121993 (111h1m37.500205028s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.631506683 (111h1m37.502589718s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.639977544 (111h1m37.511060589s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.646143775 (111h1m37.51722681s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.657520980 (111h1m37.528604015s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.659491392 (111h1m37.530574427s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.663175928 (111h1m37.534258963s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.665314009 (111h1m37.536397054s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.670795998 (111h1m37.541879033s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.679195422 (111h1m37.550278467s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.828334161 (111h1m37.699417196s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.834834629 (111h1m37.705917664s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:31:04.695992065 (111h2m4.56707511s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 10 06:31:04.696053741 (111h2m4.567136776s) [seg_pos={0 0}, upload=96845]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 06:31:04.939762863 (111h2m4.810845908s) [upload=96845]: Done uploading segments... Dec 10 06:31:04.939817447 (111h2m4.810900492s) [upload=96845]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.860608227s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [..........................X...X......X...XX...X.XXXX...X....] duration: 4.056125604s trace: https://jaeger.dev.storj.io/trace/1e2823346b2516c6 FAIL (2025-12-10 07:04:06 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.05615962s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 7.222436671s trace: https://jaeger.dev.storj.io/trace/397b46dd018f3fc5 FAIL (2025-12-10 07:03:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 06:30:35.391807620 (111h1m35.262890665s) [upload=96845]: Starting upload Dec 10 06:30:35.391896590 (111h1m35.262979655s) [upload=96845]: Done starting upload Dec 10 06:30:35.391969459 (111h1m35.263052504s) [upload=96845]: Uploading segments... Dec 10 06:30:35.392189627 (111h1m35.263272663s) [upload=96845]: Got next segment. Inline: false Dec 10 06:30:35.392217962 (111h1m35.263300997s) [upload=96845]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 06:30:35.392223091 (111h1m35.263306126s) [upload=96845]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 06:30:35.639824280 (111h1m35.510907325s) [seg_pos={0 0}, upload=96845]: Begin upload segment... Dec 10 06:30:35.639915774 (111h1m35.51099882s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.639970037 (111h1m35.511053072s) [seg_pos={0 0}, upload=96845]: Done begin upload segment. Dec 10 06:30:35.640008270 (111h1m35.511091305s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640528070 (111h1m35.511611105s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640677645 (111h1m35.51176068s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641070303 (111h1m35.512153348s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641398267 (111h1m35.512481312s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641982800 (111h1m35.513065845s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642065027 (111h1m35.513148062s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642657175 (111h1m35.51374021s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642738600 (111h1m35.513821645s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.643375131 (111h1m35.514458166s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644145648 (111h1m35.515228683s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644951062 (111h1m35.516034107s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.647301418 (111h1m35.518384453s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.648638635 (111h1m35.51972168s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.650012029 (111h1m35.521095075s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:36.662681817 (111h1m36.533764862s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.773459999 (111h1m36.644543034s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.788144411 (111h1m36.659227466s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:37.344090880 (111h1m37.215173935s) [upload=96845]: Next returned nil segment Dec 10 06:30:37.344104926 (111h1m37.215187971s) [upload=96845]: Waiting for error group managing segments... Dec 10 06:30:37.629121993 (111h1m37.500205028s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.631506683 (111h1m37.502589718s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.639977544 (111h1m37.511060589s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.646143775 (111h1m37.51722681s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.657520980 (111h1m37.528604015s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.659491392 (111h1m37.530574427s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.663175928 (111h1m37.534258963s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.665314009 (111h1m37.536397054s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.670795998 (111h1m37.541879033s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.679195422 (111h1m37.550278467s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.828334161 (111h1m37.699417196s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.834834629 (111h1m37.705917664s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:31:04.695992065 (111h2m4.56707511s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 10 06:31:04.696053741 (111h2m4.567136776s) [seg_pos={0 0}, upload=96845]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 06:31:04.939762863 (111h2m4.810845908s) [upload=96845]: Done uploading segments... Dec 10 06:31:04.939817447 (111h2m4.810900492s) [upload=96845]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.222465334s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [...........................X...X......X...XX...X.XXXX...X...] duration: 3.930781653s trace: https://jaeger.dev.storj.io/trace/1e2823346b2516c6 FAIL (2025-12-10 07:03:22 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.930818023s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.864062928s trace: https://jaeger.dev.storj.io/trace/397b46dd018f3fc5 FAIL (2025-12-10 07:03:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 06:30:35.391807620 (111h1m35.262890665s) [upload=96845]: Starting upload Dec 10 06:30:35.391896590 (111h1m35.262979655s) [upload=96845]: Done starting upload Dec 10 06:30:35.391969459 (111h1m35.263052504s) [upload=96845]: Uploading segments... Dec 10 06:30:35.392189627 (111h1m35.263272663s) [upload=96845]: Got next segment. Inline: false Dec 10 06:30:35.392217962 (111h1m35.263300997s) [upload=96845]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 06:30:35.392223091 (111h1m35.263306126s) [upload=96845]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 06:30:35.639824280 (111h1m35.510907325s) [seg_pos={0 0}, upload=96845]: Begin upload segment... Dec 10 06:30:35.639915774 (111h1m35.51099882s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.639970037 (111h1m35.511053072s) [seg_pos={0 0}, upload=96845]: Done begin upload segment. Dec 10 06:30:35.640008270 (111h1m35.511091305s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640528070 (111h1m35.511611105s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640677645 (111h1m35.51176068s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641070303 (111h1m35.512153348s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641398267 (111h1m35.512481312s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641982800 (111h1m35.513065845s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642065027 (111h1m35.513148062s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642657175 (111h1m35.51374021s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642738600 (111h1m35.513821645s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.643375131 (111h1m35.514458166s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644145648 (111h1m35.515228683s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644951062 (111h1m35.516034107s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.647301418 (111h1m35.518384453s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.648638635 (111h1m35.51972168s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.650012029 (111h1m35.521095075s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:36.662681817 (111h1m36.533764862s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.773459999 (111h1m36.644543034s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.788144411 (111h1m36.659227466s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:37.344090880 (111h1m37.215173935s) [upload=96845]: Next returned nil segment Dec 10 06:30:37.344104926 (111h1m37.215187971s) [upload=96845]: Waiting for error group managing segments... Dec 10 06:30:37.629121993 (111h1m37.500205028s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.631506683 (111h1m37.502589718s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.639977544 (111h1m37.511060589s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.646143775 (111h1m37.51722681s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.657520980 (111h1m37.528604015s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.659491392 (111h1m37.530574427s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.663175928 (111h1m37.534258963s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.665314009 (111h1m37.536397054s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.670795998 (111h1m37.541879033s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.679195422 (111h1m37.550278467s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.828334161 (111h1m37.699417196s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.834834629 (111h1m37.705917664s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:31:04.695992065 (111h2m4.56707511s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 10 06:31:04.696053741 (111h2m4.567136776s) [seg_pos={0 0}, upload=96845]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 06:31:04.939762863 (111h2m4.810845908s) [upload=96845]: Done uploading segments... Dec 10 06:31:04.939817447 (111h2m4.810900492s) [upload=96845]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.864118283s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.762801977s trace: https://jaeger.dev.storj.io/trace/397b46dd018f3fc5 FAIL (2025-12-10 07:02:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 06:30:35.391807620 (111h1m35.262890665s) [upload=96845]: Starting upload Dec 10 06:30:35.391896590 (111h1m35.262979655s) [upload=96845]: Done starting upload Dec 10 06:30:35.391969459 (111h1m35.263052504s) [upload=96845]: Uploading segments... Dec 10 06:30:35.392189627 (111h1m35.263272663s) [upload=96845]: Got next segment. Inline: false Dec 10 06:30:35.392217962 (111h1m35.263300997s) [upload=96845]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 06:30:35.392223091 (111h1m35.263306126s) [upload=96845]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 06:30:35.639824280 (111h1m35.510907325s) [seg_pos={0 0}, upload=96845]: Begin upload segment... Dec 10 06:30:35.639915774 (111h1m35.51099882s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.639970037 (111h1m35.511053072s) [seg_pos={0 0}, upload=96845]: Done begin upload segment. Dec 10 06:30:35.640008270 (111h1m35.511091305s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640528070 (111h1m35.511611105s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640677645 (111h1m35.51176068s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641070303 (111h1m35.512153348s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641398267 (111h1m35.512481312s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641982800 (111h1m35.513065845s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642065027 (111h1m35.513148062s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642657175 (111h1m35.51374021s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642738600 (111h1m35.513821645s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.643375131 (111h1m35.514458166s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644145648 (111h1m35.515228683s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644951062 (111h1m35.516034107s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.647301418 (111h1m35.518384453s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.648638635 (111h1m35.51972168s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.650012029 (111h1m35.521095075s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:36.662681817 (111h1m36.533764862s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.773459999 (111h1m36.644543034s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.788144411 (111h1m36.659227466s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:37.344090880 (111h1m37.215173935s) [upload=96845]: Next returned nil segment Dec 10 06:30:37.344104926 (111h1m37.215187971s) [upload=96845]: Waiting for error group managing segments... Dec 10 06:30:37.629121993 (111h1m37.500205028s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.631506683 (111h1m37.502589718s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.639977544 (111h1m37.511060589s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.646143775 (111h1m37.51722681s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.657520980 (111h1m37.528604015s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.659491392 (111h1m37.530574427s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.663175928 (111h1m37.534258963s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.665314009 (111h1m37.536397054s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.670795998 (111h1m37.541879033s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.679195422 (111h1m37.550278467s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.828334161 (111h1m37.699417196s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.834834629 (111h1m37.705917664s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:31:04.695992065 (111h2m4.56707511s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 10 06:31:04.696053741 (111h2m4.567136776s) [seg_pos={0 0}, upload=96845]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 06:31:04.939762863 (111h2m4.810845908s) [upload=96845]: Done uploading segments... Dec 10 06:31:04.939817447 (111h2m4.810900492s) [upload=96845]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.762833136s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [............................X...X......X...XX...X.XXXX...X..] duration: 4.098099651s trace: https://jaeger.dev.storj.io/trace/1e2823346b2516c6 FAIL (2025-12-10 07:02:45 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.098164535s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.755387465s trace: https://jaeger.dev.storj.io/trace/397b46dd018f3fc5 FAIL (2025-12-10 07:02:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 06:30:35.391807620 (111h1m35.262890665s) [upload=96845]: Starting upload Dec 10 06:30:35.391896590 (111h1m35.262979655s) [upload=96845]: Done starting upload Dec 10 06:30:35.391969459 (111h1m35.263052504s) [upload=96845]: Uploading segments... Dec 10 06:30:35.392189627 (111h1m35.263272663s) [upload=96845]: Got next segment. Inline: false Dec 10 06:30:35.392217962 (111h1m35.263300997s) [upload=96845]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 06:30:35.392223091 (111h1m35.263306126s) [upload=96845]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 06:30:35.639824280 (111h1m35.510907325s) [seg_pos={0 0}, upload=96845]: Begin upload segment... Dec 10 06:30:35.639915774 (111h1m35.51099882s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.639970037 (111h1m35.511053072s) [seg_pos={0 0}, upload=96845]: Done begin upload segment. Dec 10 06:30:35.640008270 (111h1m35.511091305s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640528070 (111h1m35.511611105s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640677645 (111h1m35.51176068s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641070303 (111h1m35.512153348s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641398267 (111h1m35.512481312s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641982800 (111h1m35.513065845s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642065027 (111h1m35.513148062s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642657175 (111h1m35.51374021s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642738600 (111h1m35.513821645s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.643375131 (111h1m35.514458166s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644145648 (111h1m35.515228683s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644951062 (111h1m35.516034107s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.647301418 (111h1m35.518384453s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.648638635 (111h1m35.51972168s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.650012029 (111h1m35.521095075s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:36.662681817 (111h1m36.533764862s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.773459999 (111h1m36.644543034s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.788144411 (111h1m36.659227466s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:37.344090880 (111h1m37.215173935s) [upload=96845]: Next returned nil segment Dec 10 06:30:37.344104926 (111h1m37.215187971s) [upload=96845]: Waiting for error group managing segments... Dec 10 06:30:37.629121993 (111h1m37.500205028s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.631506683 (111h1m37.502589718s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.639977544 (111h1m37.511060589s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.646143775 (111h1m37.51722681s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.657520980 (111h1m37.528604015s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.659491392 (111h1m37.530574427s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.663175928 (111h1m37.534258963s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.665314009 (111h1m37.536397054s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.670795998 (111h1m37.541879033s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.679195422 (111h1m37.550278467s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.828334161 (111h1m37.699417196s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.834834629 (111h1m37.705917664s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:31:04.695992065 (111h2m4.56707511s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 10 06:31:04.696053741 (111h2m4.567136776s) [seg_pos={0 0}, upload=96845]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 06:31:04.939762863 (111h2m4.810845908s) [upload=96845]: Done uploading segments... Dec 10 06:31:04.939817447 (111h2m4.810900492s) [upload=96845]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.755417141s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.838059544s trace: https://jaeger.dev.storj.io/trace/397b46dd018f3fc5 FAIL (2025-12-10 07:02:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 06:30:35.391807620 (111h1m35.262890665s) [upload=96845]: Starting upload Dec 10 06:30:35.391896590 (111h1m35.262979655s) [upload=96845]: Done starting upload Dec 10 06:30:35.391969459 (111h1m35.263052504s) [upload=96845]: Uploading segments... Dec 10 06:30:35.392189627 (111h1m35.263272663s) [upload=96845]: Got next segment. Inline: false Dec 10 06:30:35.392217962 (111h1m35.263300997s) [upload=96845]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 06:30:35.392223091 (111h1m35.263306126s) [upload=96845]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 06:30:35.639824280 (111h1m35.510907325s) [seg_pos={0 0}, upload=96845]: Begin upload segment... Dec 10 06:30:35.639915774 (111h1m35.51099882s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.639970037 (111h1m35.511053072s) [seg_pos={0 0}, upload=96845]: Done begin upload segment. Dec 10 06:30:35.640008270 (111h1m35.511091305s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640528070 (111h1m35.511611105s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640677645 (111h1m35.51176068s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641070303 (111h1m35.512153348s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641398267 (111h1m35.512481312s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641982800 (111h1m35.513065845s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642065027 (111h1m35.513148062s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642657175 (111h1m35.51374021s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642738600 (111h1m35.513821645s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.643375131 (111h1m35.514458166s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644145648 (111h1m35.515228683s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644951062 (111h1m35.516034107s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.647301418 (111h1m35.518384453s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.648638635 (111h1m35.51972168s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.650012029 (111h1m35.521095075s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:36.662681817 (111h1m36.533764862s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.773459999 (111h1m36.644543034s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.788144411 (111h1m36.659227466s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:37.344090880 (111h1m37.215173935s) [upload=96845]: Next returned nil segment Dec 10 06:30:37.344104926 (111h1m37.215187971s) [upload=96845]: Waiting for error group managing segments... Dec 10 06:30:37.629121993 (111h1m37.500205028s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.631506683 (111h1m37.502589718s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.639977544 (111h1m37.511060589s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.646143775 (111h1m37.51722681s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.657520980 (111h1m37.528604015s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.659491392 (111h1m37.530574427s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.663175928 (111h1m37.534258963s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.665314009 (111h1m37.536397054s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.670795998 (111h1m37.541879033s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.679195422 (111h1m37.550278467s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.828334161 (111h1m37.699417196s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.834834629 (111h1m37.705917664s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:31:04.695992065 (111h2m4.56707511s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 10 06:31:04.696053741 (111h2m4.567136776s) [seg_pos={0 0}, upload=96845]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 06:31:04.939762863 (111h2m4.810845908s) [upload=96845]: Done uploading segments... Dec 10 06:31:04.939817447 (111h2m4.810900492s) [upload=96845]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.83807922s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.............................X...X......X...XX...X.XXXX...X.] duration: 3.924634789s trace: https://jaeger.dev.storj.io/trace/1e2823346b2516c6 FAIL (2025-12-10 07:01:51 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.924658215s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 9.187222958s trace: https://jaeger.dev.storj.io/trace/397b46dd018f3fc5 FAIL (2025-12-10 07:01:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 06:30:35.391807620 (111h1m35.262890665s) [upload=96845]: Starting upload Dec 10 06:30:35.391896590 (111h1m35.262979655s) [upload=96845]: Done starting upload Dec 10 06:30:35.391969459 (111h1m35.263052504s) [upload=96845]: Uploading segments... Dec 10 06:30:35.392189627 (111h1m35.263272663s) [upload=96845]: Got next segment. Inline: false Dec 10 06:30:35.392217962 (111h1m35.263300997s) [upload=96845]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 06:30:35.392223091 (111h1m35.263306126s) [upload=96845]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 06:30:35.639824280 (111h1m35.510907325s) [seg_pos={0 0}, upload=96845]: Begin upload segment... Dec 10 06:30:35.639915774 (111h1m35.51099882s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.639970037 (111h1m35.511053072s) [seg_pos={0 0}, upload=96845]: Done begin upload segment. Dec 10 06:30:35.640008270 (111h1m35.511091305s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640528070 (111h1m35.511611105s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640677645 (111h1m35.51176068s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641070303 (111h1m35.512153348s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641398267 (111h1m35.512481312s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641982800 (111h1m35.513065845s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642065027 (111h1m35.513148062s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642657175 (111h1m35.51374021s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642738600 (111h1m35.513821645s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.643375131 (111h1m35.514458166s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644145648 (111h1m35.515228683s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644951062 (111h1m35.516034107s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.647301418 (111h1m35.518384453s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.648638635 (111h1m35.51972168s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.650012029 (111h1m35.521095075s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:36.662681817 (111h1m36.533764862s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.773459999 (111h1m36.644543034s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.788144411 (111h1m36.659227466s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:37.344090880 (111h1m37.215173935s) [upload=96845]: Next returned nil segment Dec 10 06:30:37.344104926 (111h1m37.215187971s) [upload=96845]: Waiting for error group managing segments... Dec 10 06:30:37.629121993 (111h1m37.500205028s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.631506683 (111h1m37.502589718s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.639977544 (111h1m37.511060589s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.646143775 (111h1m37.51722681s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.657520980 (111h1m37.528604015s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.659491392 (111h1m37.530574427s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.663175928 (111h1m37.534258963s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.665314009 (111h1m37.536397054s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.670795998 (111h1m37.541879033s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.679195422 (111h1m37.550278467s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.828334161 (111h1m37.699417196s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.834834629 (111h1m37.705917664s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:31:04.695992065 (111h2m4.56707511s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 10 06:31:04.696053741 (111h2m4.567136776s) [seg_pos={0 0}, upload=96845]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 06:31:04.939762863 (111h2m4.810845908s) [upload=96845]: Done uploading segments... Dec 10 06:31:04.939817447 (111h2m4.810900492s) [upload=96845]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.18725557s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 9.036837754s trace: https://jaeger.dev.storj.io/trace/397b46dd018f3fc5 FAIL (2025-12-10 07:00:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 06:30:35.391807620 (111h1m35.262890665s) [upload=96845]: Starting upload Dec 10 06:30:35.391896590 (111h1m35.262979655s) [upload=96845]: Done starting upload Dec 10 06:30:35.391969459 (111h1m35.263052504s) [upload=96845]: Uploading segments... Dec 10 06:30:35.392189627 (111h1m35.263272663s) [upload=96845]: Got next segment. Inline: false Dec 10 06:30:35.392217962 (111h1m35.263300997s) [upload=96845]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 06:30:35.392223091 (111h1m35.263306126s) [upload=96845]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 06:30:35.639824280 (111h1m35.510907325s) [seg_pos={0 0}, upload=96845]: Begin upload segment... Dec 10 06:30:35.639915774 (111h1m35.51099882s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.639970037 (111h1m35.511053072s) [seg_pos={0 0}, upload=96845]: Done begin upload segment. Dec 10 06:30:35.640008270 (111h1m35.511091305s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640528070 (111h1m35.511611105s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640677645 (111h1m35.51176068s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641070303 (111h1m35.512153348s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641398267 (111h1m35.512481312s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641982800 (111h1m35.513065845s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642065027 (111h1m35.513148062s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642657175 (111h1m35.51374021s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642738600 (111h1m35.513821645s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.643375131 (111h1m35.514458166s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644145648 (111h1m35.515228683s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644951062 (111h1m35.516034107s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.647301418 (111h1m35.518384453s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.648638635 (111h1m35.51972168s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.650012029 (111h1m35.521095075s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:36.662681817 (111h1m36.533764862s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.773459999 (111h1m36.644543034s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.788144411 (111h1m36.659227466s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:37.344090880 (111h1m37.215173935s) [upload=96845]: Next returned nil segment Dec 10 06:30:37.344104926 (111h1m37.215187971s) [upload=96845]: Waiting for error group managing segments... Dec 10 06:30:37.629121993 (111h1m37.500205028s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.631506683 (111h1m37.502589718s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.639977544 (111h1m37.511060589s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.646143775 (111h1m37.51722681s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.657520980 (111h1m37.528604015s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.659491392 (111h1m37.530574427s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.663175928 (111h1m37.534258963s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.665314009 (111h1m37.536397054s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.670795998 (111h1m37.541879033s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.679195422 (111h1m37.550278467s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.828334161 (111h1m37.699417196s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.834834629 (111h1m37.705917664s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:31:04.695992065 (111h2m4.56707511s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 10 06:31:04.696053741 (111h2m4.567136776s) [seg_pos={0 0}, upload=96845]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 06:31:04.939762863 (111h2m4.810845908s) [upload=96845]: Done uploading segments... Dec 10 06:31:04.939817447 (111h2m4.810900492s) [upload=96845]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.036865396s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [..............................X...X......X...XX...X.XXXX...X] duration: 5.001914654s trace: https://jaeger.dev.storj.io/trace/1e2823346b2516c6 FAIL (2025-12-10 07:00:51 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00194433s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 8.861240857s trace: https://jaeger.dev.storj.io/trace/397b46dd018f3fc5 FAIL (2025-12-10 07:00:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 06:30:35.391807620 (111h1m35.262890665s) [upload=96845]: Starting upload Dec 10 06:30:35.391896590 (111h1m35.262979655s) [upload=96845]: Done starting upload Dec 10 06:30:35.391969459 (111h1m35.263052504s) [upload=96845]: Uploading segments... Dec 10 06:30:35.392189627 (111h1m35.263272663s) [upload=96845]: Got next segment. Inline: false Dec 10 06:30:35.392217962 (111h1m35.263300997s) [upload=96845]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 06:30:35.392223091 (111h1m35.263306126s) [upload=96845]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 06:30:35.639824280 (111h1m35.510907325s) [seg_pos={0 0}, upload=96845]: Begin upload segment... Dec 10 06:30:35.639915774 (111h1m35.51099882s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.639970037 (111h1m35.511053072s) [seg_pos={0 0}, upload=96845]: Done begin upload segment. Dec 10 06:30:35.640008270 (111h1m35.511091305s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640528070 (111h1m35.511611105s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640677645 (111h1m35.51176068s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641070303 (111h1m35.512153348s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641398267 (111h1m35.512481312s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641982800 (111h1m35.513065845s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642065027 (111h1m35.513148062s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642657175 (111h1m35.51374021s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642738600 (111h1m35.513821645s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.643375131 (111h1m35.514458166s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644145648 (111h1m35.515228683s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644951062 (111h1m35.516034107s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.647301418 (111h1m35.518384453s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.648638635 (111h1m35.51972168s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.650012029 (111h1m35.521095075s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:36.662681817 (111h1m36.533764862s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.773459999 (111h1m36.644543034s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.788144411 (111h1m36.659227466s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:37.344090880 (111h1m37.215173935s) [upload=96845]: Next returned nil segment Dec 10 06:30:37.344104926 (111h1m37.215187971s) [upload=96845]: Waiting for error group managing segments... Dec 10 06:30:37.629121993 (111h1m37.500205028s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.631506683 (111h1m37.502589718s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.639977544 (111h1m37.511060589s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.646143775 (111h1m37.51722681s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.657520980 (111h1m37.528604015s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.659491392 (111h1m37.530574427s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.663175928 (111h1m37.534258963s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.665314009 (111h1m37.536397054s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.670795998 (111h1m37.541879033s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.679195422 (111h1m37.550278467s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.828334161 (111h1m37.699417196s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.834834629 (111h1m37.705917664s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:31:04.695992065 (111h2m4.56707511s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 10 06:31:04.696053741 (111h2m4.567136776s) [seg_pos={0 0}, upload=96845]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 06:31:04.939762863 (111h2m4.810845908s) [upload=96845]: Done uploading segments... Dec 10 06:31:04.939817447 (111h2m4.810900492s) [upload=96845]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.861276675s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [...............................X...X......X...XX...X.XXXX...] duration: 4.053764359s trace: https://jaeger.dev.storj.io/trace/5b4026669c30c3bd FAIL (2025-12-10 07:00:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.053800928s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [................................X...X......X...XX...X.XXXX..] duration: 4.115415161s trace: https://jaeger.dev.storj.io/trace/5b4026669c30c3bd FAIL (2025-12-10 06:59:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.115445048s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 8.355831988s trace: https://jaeger.dev.storj.io/trace/397b46dd018f3fc5 FAIL (2025-12-10 06:59:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 06:30:35.391807620 (111h1m35.262890665s) [upload=96845]: Starting upload Dec 10 06:30:35.391896590 (111h1m35.262979655s) [upload=96845]: Done starting upload Dec 10 06:30:35.391969459 (111h1m35.263052504s) [upload=96845]: Uploading segments... Dec 10 06:30:35.392189627 (111h1m35.263272663s) [upload=96845]: Got next segment. Inline: false Dec 10 06:30:35.392217962 (111h1m35.263300997s) [upload=96845]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 06:30:35.392223091 (111h1m35.263306126s) [upload=96845]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 06:30:35.639824280 (111h1m35.510907325s) [seg_pos={0 0}, upload=96845]: Begin upload segment... Dec 10 06:30:35.639915774 (111h1m35.51099882s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.639970037 (111h1m35.511053072s) [seg_pos={0 0}, upload=96845]: Done begin upload segment. Dec 10 06:30:35.640008270 (111h1m35.511091305s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640528070 (111h1m35.511611105s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640677645 (111h1m35.51176068s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641070303 (111h1m35.512153348s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641398267 (111h1m35.512481312s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641982800 (111h1m35.513065845s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642065027 (111h1m35.513148062s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642657175 (111h1m35.51374021s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642738600 (111h1m35.513821645s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.643375131 (111h1m35.514458166s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644145648 (111h1m35.515228683s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644951062 (111h1m35.516034107s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.647301418 (111h1m35.518384453s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.648638635 (111h1m35.51972168s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.650012029 (111h1m35.521095075s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:36.662681817 (111h1m36.533764862s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.773459999 (111h1m36.644543034s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.788144411 (111h1m36.659227466s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:37.344090880 (111h1m37.215173935s) [upload=96845]: Next returned nil segment Dec 10 06:30:37.344104926 (111h1m37.215187971s) [upload=96845]: Waiting for error group managing segments... Dec 10 06:30:37.629121993 (111h1m37.500205028s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.631506683 (111h1m37.502589718s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.639977544 (111h1m37.511060589s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.646143775 (111h1m37.51722681s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.657520980 (111h1m37.528604015s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.659491392 (111h1m37.530574427s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.663175928 (111h1m37.534258963s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.665314009 (111h1m37.536397054s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.670795998 (111h1m37.541879033s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.679195422 (111h1m37.550278467s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.828334161 (111h1m37.699417196s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.834834629 (111h1m37.705917664s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:31:04.695992065 (111h2m4.56707511s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 10 06:31:04.696053741 (111h2m4.567136776s) [seg_pos={0 0}, upload=96845]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 06:31:04.939762863 (111h2m4.810845908s) [upload=96845]: Done uploading segments... Dec 10 06:31:04.939817447 (111h2m4.810900492s) [upload=96845]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.35586453s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.................................X...X......X...XX...X.XXXX.] duration: 4.088051886s trace: https://jaeger.dev.storj.io/trace/5b4026669c30c3bd FAIL (2025-12-10 06:59:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.088084138s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 7.975225342s trace: https://jaeger.dev.storj.io/trace/397b46dd018f3fc5 FAIL (2025-12-10 06:59:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 06:30:35.391807620 (111h1m35.262890665s) [upload=96845]: Starting upload Dec 10 06:30:35.391896590 (111h1m35.262979655s) [upload=96845]: Done starting upload Dec 10 06:30:35.391969459 (111h1m35.263052504s) [upload=96845]: Uploading segments... Dec 10 06:30:35.392189627 (111h1m35.263272663s) [upload=96845]: Got next segment. Inline: false Dec 10 06:30:35.392217962 (111h1m35.263300997s) [upload=96845]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 06:30:35.392223091 (111h1m35.263306126s) [upload=96845]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 06:30:35.639824280 (111h1m35.510907325s) [seg_pos={0 0}, upload=96845]: Begin upload segment... Dec 10 06:30:35.639915774 (111h1m35.51099882s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.639970037 (111h1m35.511053072s) [seg_pos={0 0}, upload=96845]: Done begin upload segment. Dec 10 06:30:35.640008270 (111h1m35.511091305s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640528070 (111h1m35.511611105s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640677645 (111h1m35.51176068s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641070303 (111h1m35.512153348s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641398267 (111h1m35.512481312s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641982800 (111h1m35.513065845s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642065027 (111h1m35.513148062s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642657175 (111h1m35.51374021s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642738600 (111h1m35.513821645s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.643375131 (111h1m35.514458166s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644145648 (111h1m35.515228683s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644951062 (111h1m35.516034107s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.647301418 (111h1m35.518384453s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.648638635 (111h1m35.51972168s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.650012029 (111h1m35.521095075s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:36.662681817 (111h1m36.533764862s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.773459999 (111h1m36.644543034s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.788144411 (111h1m36.659227466s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:37.344090880 (111h1m37.215173935s) [upload=96845]: Next returned nil segment Dec 10 06:30:37.344104926 (111h1m37.215187971s) [upload=96845]: Waiting for error group managing segments... Dec 10 06:30:37.629121993 (111h1m37.500205028s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.631506683 (111h1m37.502589718s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.639977544 (111h1m37.511060589s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.646143775 (111h1m37.51722681s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.657520980 (111h1m37.528604015s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.659491392 (111h1m37.530574427s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.663175928 (111h1m37.534258963s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.665314009 (111h1m37.536397054s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.670795998 (111h1m37.541879033s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.679195422 (111h1m37.550278467s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.828334161 (111h1m37.699417196s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.834834629 (111h1m37.705917664s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:31:04.695992065 (111h2m4.56707511s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 10 06:31:04.696053741 (111h2m4.567136776s) [seg_pos={0 0}, upload=96845]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 06:31:04.939762863 (111h2m4.810845908s) [upload=96845]: Done uploading segments... Dec 10 06:31:04.939817447 (111h2m4.810900492s) [upload=96845]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.975243526s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [..................................X...X......X...XX...X.XXXX] duration: 5.001293631s trace: https://jaeger.dev.storj.io/trace/5b4026669c30c3bd FAIL (2025-12-10 06:58:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001320092s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.566938586s trace: https://jaeger.dev.storj.io/trace/397b46dd018f3fc5 FAIL (2025-12-10 06:58:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 06:30:35.391807620 (111h1m35.262890665s) [upload=96845]: Starting upload Dec 10 06:30:35.391896590 (111h1m35.262979655s) [upload=96845]: Done starting upload Dec 10 06:30:35.391969459 (111h1m35.263052504s) [upload=96845]: Uploading segments... Dec 10 06:30:35.392189627 (111h1m35.263272663s) [upload=96845]: Got next segment. Inline: false Dec 10 06:30:35.392217962 (111h1m35.263300997s) [upload=96845]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 06:30:35.392223091 (111h1m35.263306126s) [upload=96845]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 06:30:35.639824280 (111h1m35.510907325s) [seg_pos={0 0}, upload=96845]: Begin upload segment... Dec 10 06:30:35.639915774 (111h1m35.51099882s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.639970037 (111h1m35.511053072s) [seg_pos={0 0}, upload=96845]: Done begin upload segment. Dec 10 06:30:35.640008270 (111h1m35.511091305s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640528070 (111h1m35.511611105s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640677645 (111h1m35.51176068s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641070303 (111h1m35.512153348s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641398267 (111h1m35.512481312s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641982800 (111h1m35.513065845s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642065027 (111h1m35.513148062s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642657175 (111h1m35.51374021s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642738600 (111h1m35.513821645s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.643375131 (111h1m35.514458166s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644145648 (111h1m35.515228683s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644951062 (111h1m35.516034107s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.647301418 (111h1m35.518384453s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.648638635 (111h1m35.51972168s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.650012029 (111h1m35.521095075s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:36.662681817 (111h1m36.533764862s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.773459999 (111h1m36.644543034s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.788144411 (111h1m36.659227466s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:37.344090880 (111h1m37.215173935s) [upload=96845]: Next returned nil segment Dec 10 06:30:37.344104926 (111h1m37.215187971s) [upload=96845]: Waiting for error group managing segments... Dec 10 06:30:37.629121993 (111h1m37.500205028s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.631506683 (111h1m37.502589718s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.639977544 (111h1m37.511060589s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.646143775 (111h1m37.51722681s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.657520980 (111h1m37.528604015s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.659491392 (111h1m37.530574427s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.663175928 (111h1m37.534258963s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.665314009 (111h1m37.536397054s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.670795998 (111h1m37.541879033s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.679195422 (111h1m37.550278467s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.828334161 (111h1m37.699417196s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.834834629 (111h1m37.705917664s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:31:04.695992065 (111h2m4.56707511s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 10 06:31:04.696053741 (111h2m4.567136776s) [seg_pos={0 0}, upload=96845]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 06:31:04.939762863 (111h2m4.810845908s) [upload=96845]: Done uploading segments... Dec 10 06:31:04.939817447 (111h2m4.810900492s) [upload=96845]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.566964426s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [...................................X...X......X...XX...X.XXX] duration: 5.001549209s trace: https://jaeger.dev.storj.io/trace/3770ce7408dfb40b FAIL (2025-12-10 06:58:34 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001578525s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [....................................X...X......X...XX...X.XX] duration: 5.001045159s trace: https://jaeger.dev.storj.io/trace/69c8300dca180913 FAIL (2025-12-10 06:58:19 UTC) orbiter warning: context deadline exceeded Duration: 5.001069347s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 8.462386992s trace: https://jaeger.dev.storj.io/trace/397b46dd018f3fc5 FAIL (2025-12-10 06:58:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 06:30:35.391807620 (111h1m35.262890665s) [upload=96845]: Starting upload Dec 10 06:30:35.391896590 (111h1m35.262979655s) [upload=96845]: Done starting upload Dec 10 06:30:35.391969459 (111h1m35.263052504s) [upload=96845]: Uploading segments... Dec 10 06:30:35.392189627 (111h1m35.263272663s) [upload=96845]: Got next segment. Inline: false Dec 10 06:30:35.392217962 (111h1m35.263300997s) [upload=96845]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 06:30:35.392223091 (111h1m35.263306126s) [upload=96845]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 06:30:35.639824280 (111h1m35.510907325s) [seg_pos={0 0}, upload=96845]: Begin upload segment... Dec 10 06:30:35.639915774 (111h1m35.51099882s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.639970037 (111h1m35.511053072s) [seg_pos={0 0}, upload=96845]: Done begin upload segment. Dec 10 06:30:35.640008270 (111h1m35.511091305s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640528070 (111h1m35.511611105s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640677645 (111h1m35.51176068s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641070303 (111h1m35.512153348s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641398267 (111h1m35.512481312s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641982800 (111h1m35.513065845s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642065027 (111h1m35.513148062s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642657175 (111h1m35.51374021s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642738600 (111h1m35.513821645s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.643375131 (111h1m35.514458166s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644145648 (111h1m35.515228683s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644951062 (111h1m35.516034107s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.647301418 (111h1m35.518384453s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.648638635 (111h1m35.51972168s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.650012029 (111h1m35.521095075s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:36.662681817 (111h1m36.533764862s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.773459999 (111h1m36.644543034s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.788144411 (111h1m36.659227466s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:37.344090880 (111h1m37.215173935s) [upload=96845]: Next returned nil segment Dec 10 06:30:37.344104926 (111h1m37.215187971s) [upload=96845]: Waiting for error group managing segments... Dec 10 06:30:37.629121993 (111h1m37.500205028s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.631506683 (111h1m37.502589718s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.639977544 (111h1m37.511060589s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.646143775 (111h1m37.51722681s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.657520980 (111h1m37.528604015s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.659491392 (111h1m37.530574427s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.663175928 (111h1m37.534258963s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.665314009 (111h1m37.536397054s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.670795998 (111h1m37.541879033s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.679195422 (111h1m37.550278467s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.828334161 (111h1m37.699417196s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.834834629 (111h1m37.705917664s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:31:04.695992065 (111h2m4.56707511s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 10 06:31:04.696053741 (111h2m4.567136776s) [seg_pos={0 0}, upload=96845]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 06:31:04.939762863 (111h2m4.810845908s) [upload=96845]: Done uploading segments... Dec 10 06:31:04.939817447 (111h2m4.810900492s) [upload=96845]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.46241731s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.....................................X...X......X...XX...X.X] duration: 5.001467445s trace: https://jaeger.dev.storj.io/trace/5dd6345e979ebeef FAIL (2025-12-10 06:57:51 UTC) orbiter warning: context deadline exceeded Duration: 5.00150675s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 7.529238311s trace: https://jaeger.dev.storj.io/trace/397b46dd018f3fc5 FAIL (2025-12-10 06:57:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 06:30:35.391807620 (111h1m35.262890665s) [upload=96845]: Starting upload Dec 10 06:30:35.391896590 (111h1m35.262979655s) [upload=96845]: Done starting upload Dec 10 06:30:35.391969459 (111h1m35.263052504s) [upload=96845]: Uploading segments... Dec 10 06:30:35.392189627 (111h1m35.263272663s) [upload=96845]: Got next segment. Inline: false Dec 10 06:30:35.392217962 (111h1m35.263300997s) [upload=96845]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 06:30:35.392223091 (111h1m35.263306126s) [upload=96845]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 06:30:35.639824280 (111h1m35.510907325s) [seg_pos={0 0}, upload=96845]: Begin upload segment... Dec 10 06:30:35.639915774 (111h1m35.51099882s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.639970037 (111h1m35.511053072s) [seg_pos={0 0}, upload=96845]: Done begin upload segment. Dec 10 06:30:35.640008270 (111h1m35.511091305s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640528070 (111h1m35.511611105s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640677645 (111h1m35.51176068s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641070303 (111h1m35.512153348s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641398267 (111h1m35.512481312s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641982800 (111h1m35.513065845s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642065027 (111h1m35.513148062s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642657175 (111h1m35.51374021s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642738600 (111h1m35.513821645s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.643375131 (111h1m35.514458166s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644145648 (111h1m35.515228683s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644951062 (111h1m35.516034107s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.647301418 (111h1m35.518384453s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.648638635 (111h1m35.51972168s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.650012029 (111h1m35.521095075s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:36.662681817 (111h1m36.533764862s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.773459999 (111h1m36.644543034s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.788144411 (111h1m36.659227466s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:37.344090880 (111h1m37.215173935s) [upload=96845]: Next returned nil segment Dec 10 06:30:37.344104926 (111h1m37.215187971s) [upload=96845]: Waiting for error group managing segments... Dec 10 06:30:37.629121993 (111h1m37.500205028s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.631506683 (111h1m37.502589718s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.639977544 (111h1m37.511060589s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.646143775 (111h1m37.51722681s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.657520980 (111h1m37.528604015s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.659491392 (111h1m37.530574427s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.663175928 (111h1m37.534258963s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.665314009 (111h1m37.536397054s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.670795998 (111h1m37.541879033s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.679195422 (111h1m37.550278467s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.828334161 (111h1m37.699417196s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.834834629 (111h1m37.705917664s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:31:04.695992065 (111h2m4.56707511s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 10 06:31:04.696053741 (111h2m4.567136776s) [seg_pos={0 0}, upload=96845]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 06:31:04.939762863 (111h2m4.810845908s) [upload=96845]: Done uploading segments... Dec 10 06:31:04.939817447 (111h2m4.810900492s) [upload=96845]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.529273919s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [......................................X...X......X...XX...X.] duration: 4.245821592s trace: https://jaeger.dev.storj.io/trace/7480b8c36cbc0ea8 FAIL (2025-12-10 06:57:02 UTC) orbiter warning: context deadline exceeded Duration: 4.245853362s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 7.094308473s trace: https://jaeger.dev.storj.io/trace/397b46dd018f3fc5 FAIL (2025-12-10 06:56:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 06:30:35.391807620 (111h1m35.262890665s) [upload=96845]: Starting upload Dec 10 06:30:35.391896590 (111h1m35.262979655s) [upload=96845]: Done starting upload Dec 10 06:30:35.391969459 (111h1m35.263052504s) [upload=96845]: Uploading segments... Dec 10 06:30:35.392189627 (111h1m35.263272663s) [upload=96845]: Got next segment. Inline: false Dec 10 06:30:35.392217962 (111h1m35.263300997s) [upload=96845]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 06:30:35.392223091 (111h1m35.263306126s) [upload=96845]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 06:30:35.639824280 (111h1m35.510907325s) [seg_pos={0 0}, upload=96845]: Begin upload segment... Dec 10 06:30:35.639915774 (111h1m35.51099882s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.639970037 (111h1m35.511053072s) [seg_pos={0 0}, upload=96845]: Done begin upload segment. Dec 10 06:30:35.640008270 (111h1m35.511091305s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640528070 (111h1m35.511611105s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640677645 (111h1m35.51176068s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641070303 (111h1m35.512153348s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641398267 (111h1m35.512481312s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641982800 (111h1m35.513065845s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642065027 (111h1m35.513148062s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642657175 (111h1m35.51374021s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642738600 (111h1m35.513821645s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.643375131 (111h1m35.514458166s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644145648 (111h1m35.515228683s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644951062 (111h1m35.516034107s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.647301418 (111h1m35.518384453s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.648638635 (111h1m35.51972168s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.650012029 (111h1m35.521095075s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:36.662681817 (111h1m36.533764862s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.773459999 (111h1m36.644543034s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.788144411 (111h1m36.659227466s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:37.344090880 (111h1m37.215173935s) [upload=96845]: Next returned nil segment Dec 10 06:30:37.344104926 (111h1m37.215187971s) [upload=96845]: Waiting for error group managing segments... Dec 10 06:30:37.629121993 (111h1m37.500205028s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.631506683 (111h1m37.502589718s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.639977544 (111h1m37.511060589s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.646143775 (111h1m37.51722681s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.657520980 (111h1m37.528604015s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.659491392 (111h1m37.530574427s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.663175928 (111h1m37.534258963s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.665314009 (111h1m37.536397054s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.670795998 (111h1m37.541879033s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.679195422 (111h1m37.550278467s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.828334161 (111h1m37.699417196s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.834834629 (111h1m37.705917664s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:31:04.695992065 (111h2m4.56707511s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 10 06:31:04.696053741 (111h2m4.567136776s) [seg_pos={0 0}, upload=96845]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 06:31:04.939762863 (111h2m4.810845908s) [upload=96845]: Done uploading segments... Dec 10 06:31:04.939817447 (111h2m4.810900492s) [upload=96845]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.09433763s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.......................................X...X......X...XX...X] duration: 5.001224008s trace: https://jaeger.dev.storj.io/trace/7480b8c36cbc0ea8 FAIL (2025-12-10 06:56:21 UTC) orbiter warning: context deadline exceeded Duration: 5.001248207s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.998695523s trace: https://jaeger.dev.storj.io/trace/397b46dd018f3fc5 FAIL (2025-12-10 06:55:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 06:30:35.391807620 (111h1m35.262890665s) [upload=96845]: Starting upload Dec 10 06:30:35.391896590 (111h1m35.262979655s) [upload=96845]: Done starting upload Dec 10 06:30:35.391969459 (111h1m35.263052504s) [upload=96845]: Uploading segments... Dec 10 06:30:35.392189627 (111h1m35.263272663s) [upload=96845]: Got next segment. Inline: false Dec 10 06:30:35.392217962 (111h1m35.263300997s) [upload=96845]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 06:30:35.392223091 (111h1m35.263306126s) [upload=96845]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 06:30:35.639824280 (111h1m35.510907325s) [seg_pos={0 0}, upload=96845]: Begin upload segment... Dec 10 06:30:35.639915774 (111h1m35.51099882s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.639970037 (111h1m35.511053072s) [seg_pos={0 0}, upload=96845]: Done begin upload segment. Dec 10 06:30:35.640008270 (111h1m35.511091305s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640528070 (111h1m35.511611105s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.640677645 (111h1m35.51176068s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641070303 (111h1m35.512153348s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641398267 (111h1m35.512481312s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.641982800 (111h1m35.513065845s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642065027 (111h1m35.513148062s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642657175 (111h1m35.51374021s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.642738600 (111h1m35.513821645s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.643375131 (111h1m35.514458166s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644145648 (111h1m35.515228683s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.644951062 (111h1m35.516034107s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.647301418 (111h1m35.518384453s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.648638635 (111h1m35.51972168s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:35.650012029 (111h1m35.521095075s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Uploading piece... Dec 10 06:30:36.662681817 (111h1m36.533764862s) [noise=true , address=145.79.227.16:4910 , piece_id=74JVZ6STFEFIUKPRQP7FQ3NALLAECOK6NCZUI42Z4OELVYUVIZGA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.773459999 (111h1m36.644543034s) [noise=true , address=77.90.31.16:4734 , piece_id=WHHY5M3QJG34D62LYXV2TDIQUGLY5L2AJNI2Y6D2YGMQ5J3EQ2HA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:36.788144411 (111h1m36.659227466s) [noise=true , address=77.90.31.15:4698 , piece_id=EU4QPBHVPW3K7W3ILINAEQIWUJCDNZS3P4677E2X5AVUMCE62JFQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 10 06:30:37.344090880 (111h1m37.215173935s) [upload=96845]: Next returned nil segment Dec 10 06:30:37.344104926 (111h1m37.215187971s) [upload=96845]: Waiting for error group managing segments... Dec 10 06:30:37.629121993 (111h1m37.500205028s) [noise=true , address=145.79.227.31:4799 , piece_id=REODEFY7RYIVOE6DNYLMVJD3WHGBHF6VJH7RFB54R2LUM4TSDU3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.631506683 (111h1m37.502589718s) [noise=true , address=145.79.227.41:4086 , piece_id=37QG2AYHNVFBECGR4IJWJNE62CCWPRIWW25DBLXMSDSLGTRJASPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.639977544 (111h1m37.511060589s) [noise=true , address=145.79.227.18:4006 , piece_id=7TVDNDR46H3XGHNMXJ6FLTOGTIJLISNCYG2TJBECGEY2D4JWIAEA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.646143775 (111h1m37.51722681s) [noise=true , address=145.79.227.21:4475 , piece_id=VREIWSZQYJ47UEONDVNJ5RIAK6FGYTYLVEUINSKHQJBKC4D5JO3Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.657520980 (111h1m37.528604015s) [noise=true , address=145.79.227.26:4654 , piece_id=NHSW3VZJZPZUH2TRXR43VNSPYMW3AFVNMH54DGYNQPMCD5CWLRLQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.659491392 (111h1m37.530574427s) [noise=true , address=145.79.227.14:4838 , piece_id=OAIVEBR3YAWMXGF6DE4E2KYW3UQJ3UGHYLWOEOK2UVARVRHJ6GVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.663175928 (111h1m37.534258963s) [noise=true , address=145.79.227.27:4689 , piece_id=RQPZ7CUZZ7AJR52P2KW7GLTP4RVMKTGR5MGHWHWZZKELKL62S3SQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.665314009 (111h1m37.536397054s) [noise=true , address=145.79.227.38:4978 , piece_id=ZKKADOHHYXFYFMWVGY4APNGBN2IANIUK5BANETV2ZYJ5XTIGZF4Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.670795998 (111h1m37.541879033s) [noise=true , address=145.79.227.33:4869 , piece_id=LDWPGC5NPZYDIDCGDD4BZRCBNWIWEBJ42JF42DITD5FCMUTMA5BA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.679195422 (111h1m37.550278467s) [noise=true , address=145.79.227.40:4051 , piece_id=KRKMZCHIQVQBEDMS2AJR4ONHJABFKDLYJWSWALHWK7VZ2BQI5I2Q, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.828334161 (111h1m37.699417196s) [noise=true , address=145.79.227.29:4943 , piece_id=GMIK3KXML2CVRBW4NXXJMAI25PPJFUPXLM6GID7Y3QFKKKYL5GPQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:30:37.834834629 (111h1m37.705917664s) [noise=true , address=77.90.31.10:4520 , piece_id=JTRPUEVXJUQPXKBD3P23DBCGJHHZ4UITRM2JDEUTD4KYIURRHCVQ, seg_pos={0 0}, upload=96845]: Done uploading piece. err: <nil> Dec 10 06:31:04.695992065 (111h2m4.56707511s) [noise=true , address=77.90.31.13:4627 , piece_id=GUNMIA6WDYEBUJM7V5RAKJFMTLNE4O2CRIEDII3BMIRENTJM4DDA, seg_pos={0 0}, upload=96845]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 10 06:31:04.696053741 (111h2m4.567136776s) [seg_pos={0 0}, upload=96845]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 06:31:04.939762863 (111h2m4.810845908s) [upload=96845]: Done uploading segments... Dec 10 06:31:04.939817447 (111h2m4.810900492s) [upload=96845]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.998739785s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [........................................X...X......X...XX...] duration: 4.301763532s trace: https://jaeger.dev.storj.io/trace/15c76ec41657c3bf FAIL (2025-12-10 06:55:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.301790744s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)