Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 3.510348038s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 19:45:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.510379226s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.743031602s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 19:44:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.743063513s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 3.541587776s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 19:44:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.54162195s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.12758363s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 19:44:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.127615422s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.639085303s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 19:43:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.639114299s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 3.571085383s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 19:43:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.571116132s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.066319642s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 19:42:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.066352034s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 3.53174167s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 19:42:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.531808417s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.151722976s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 19:41:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.151781928s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 3.488871709s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 19:41:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.488899391s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................X............................] duration: 3.421245035s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 19:40:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.421277437s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.945841957s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 19:40:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.945871584s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.776898208s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 19:39:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.776941041s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................X...........................] duration: 3.430391979s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 19:39:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.430513s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.795342966s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 19:39:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.795369636s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................X..........................] duration: 3.469915172s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 19:39:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.470120503s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.008426588s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 19:39:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.008473618s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.980123813s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 19:38:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.980150523s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................X.........................] duration: 3.508464495s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 19:38:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.508490905s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................X........................] duration: 3.611924076s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 19:37:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.611950607s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.987774491s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 19:37:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.987804417s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................X.......................] duration: 3.466003939s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 19:37:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.466065205s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................X......................] duration: 3.41930427s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 19:37:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.419334689s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.802244998s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 19:37:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.802277389s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................X.....................] duration: 3.533998717s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 19:36:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.534029064s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.771166809s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 19:36:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.771195032s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................X....................] duration: 3.506697899s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 19:36:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.50672983s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................X...................] duration: 3.427313091s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 19:35:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.427356543s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.974263357s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 19:35:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.974287943s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................X..................] duration: 3.406989042s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 19:35:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.407015672s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................X.................] duration: 3.425433283s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 19:34:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.425464141s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.915359083s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 19:34:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.915398318s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................X................] duration: 3.510714402s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 19:34:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.510743819s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 3.515736886s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 19:34:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.515765411s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.881313935s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 19:33:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.881334654s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 3.514705313s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 19:33:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.514739437s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 3.513292361s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 19:32:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.513312829s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.853257281s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 19:32:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.853286756s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.932918299s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 19:31:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.932952645s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 3.448514664s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 19:31:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.448537156s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.707533476s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 19:30:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.70756086s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 4.450874494s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 19:30:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.450899732s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 3.516700793s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 19:30:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.516731421s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.837847588s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 19:29:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.83787462s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.987509329s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 19:29:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.987543735s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 3.427504045s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 19:28:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.427547569s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.952259287s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 19:28:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.952287802s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 3.547145042s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 19:28:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.547176701s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.959296307s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 19:27:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.95933552s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 3.396402913s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 19:27:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.39642734s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)