Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............XX..............................................] duration: 3.403496242s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:36:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.403525167s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............XX.............................................] duration: 3.481973635s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:36:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.482200807s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.907925797s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:35:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.907954301s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............XX............................................] duration: 3.572634129s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:34:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.572660429s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.040778501s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:34:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.040797327s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............XX...........................................] duration: 3.53583647s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:34:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.535868001s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.742759251s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:33:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.742789049s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.695802939s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:33:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.695859276s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................XX..........................................] duration: 3.477176271s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:33:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.477207761s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................XX.........................................] duration: 4.330893707s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:32:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.330926608s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.744187407s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:32:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.744215011s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................XX........................................] duration: 4.41975276s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:32:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.419769732s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.785080127s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:31:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.785107229s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................XX.......................................] duration: 3.471478291s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:31:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.47149303s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.872001391s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:31:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.872034013s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................XX......................................] duration: 3.536836985s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:30:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.536870569s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.791489751s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:30:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.791521211s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....................XX.....................................] duration: 3.543291254s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:30:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.54339464s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.964577793s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:29:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.964600826s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......................XX....................................] duration: 3.376723392s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:29:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.376925528s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.886144936s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:29:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.886165356s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................XX...................................] duration: 3.44686207s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:28:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.446888009s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.794799611s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:28:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.794829077s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.731914095s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:28:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.731945455s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................XX..................................] duration: 3.503817644s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:27:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.503849034s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.857123519s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:27:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.857148718s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................XX.................................] duration: 3.59958471s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:26:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.599614045s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.80568564s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:26:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.805714766s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................XX................................] duration: 3.430662262s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:25:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.430688692s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.116043304s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:25:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.11607881s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........................XX...............................] duration: 3.511359514s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:25:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.511399181s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.102741212s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:24:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.102775398s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............................XX..............................] duration: 3.530963473s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:24:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.530990154s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.915803562s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:23:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.915829211s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............................XX.............................] duration: 3.616628041s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:23:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.616654511s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.909031397s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:22:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.909059231s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.920901409s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:22:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.920949241s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............................XX............................] duration: 3.575223329s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:22:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.575252784s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.748943378s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:21:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.748972855s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.705771967s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:21:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.705804049s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............................XX...........................] duration: 3.548895909s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:21:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.548918801s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 6.002273821s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:20:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.002300261s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................................XX..........................] duration: 3.34226875s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:19:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.342297816s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.822805468s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:19:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.822824286s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 1.702030441s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 03:19:30 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 1.702060598s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................................XX.........................] duration: 3.516435639s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:19:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.516459193s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.199339686s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 03:19:12 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.199365545s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.958759378s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:18:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.958795297s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................................XX........................] duration: 3.532566555s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:18:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.532591702s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.85643767s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:18:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.856466916s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)