Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................................XX.......................] duration: 3.440580469s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:17:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.440605106s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.153994604s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 03:17:15 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.154019362s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.721071962s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:16:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.721096419s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.749201364s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:16:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.749231261s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.247915452s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 03:16:29 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.247930692s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................................XX......................] duration: 3.604649222s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:16:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.604676515s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.780484909s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:16:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.780525176s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.173887903s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 03:15:43 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.173915376s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....................................XX.....................] duration: 3.578559525s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:15:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.578585774s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.84073466s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:15:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.84077196s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.259053733s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 03:15:02 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.25908875s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.198466365s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 03:14:34 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.19849524s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......................................XX....................] duration: 3.522269418s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:14:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.522300627s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.905785626s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:14:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.905816935s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................................XX...................] duration: 3.611592002s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:14:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.611612411s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.157209964s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 03:13:47 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.157291078s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.759223912s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:13:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.759331726s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................................XX..................] duration: 3.484407494s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:13:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.484431559s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.959114063s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:12:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.959162364s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.253642018s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 03:12:39 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.253692454s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................................XX.................] duration: 3.579436687s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:12:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.57952264s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.806865316s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:11:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.806885103s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................................XX................] duration: 3.540000001s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:11:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.540028665s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 2.24106804s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 03:11:27 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.241142513s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.262578671s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 03:10:52 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.262608828s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.001883822s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:10:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.001915753s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........................................XX...............] duration: 4.367881955s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:10:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.367949974s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.94843779s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:10:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.948471594s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 2.1799383s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 03:10:14 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.17997016s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............................................XX..............] duration: 3.684413859s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:10:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.684568773s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.941549013s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:10:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.941605791s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.182031006s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 03:09:54 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.182057867s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.758331105s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:09:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.758354329s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.179689335s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 03:09:38 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.179725103s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............................................XX.............] duration: 3.425591027s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:09:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.425618068s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............................................XX............] duration: 3.464102756s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:09:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.464130629s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.159438767s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 03:09:14 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.159468332s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.15341246s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 03:08:53 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.153442087s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............................................XX...........] duration: 3.470596926s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:08:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.470626633s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.902833982s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:08:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.902858951s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.197216701s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 03:08:30 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.197254904s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................................................XX..........] duration: 3.429002012s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:08:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.429036628s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.848592859s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:07:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.848716454s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................................................XX.........] duration: 3.539297513s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:07:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.539330425s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.213856484s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 03:07:24 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.213889736s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................................................XX........] duration: 3.484762866s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:07:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.48479147s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.751296612s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:06:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.751320918s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.239428379s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 03:06:19 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.239457934s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.976513281s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:06:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.976551926s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................................................XX.......] duration: 3.380729383s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:06:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.380782834s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)