Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.12366273s trace: https://jaeger.dev.storj.io/trace/1b48011ff2f3bc78 FAIL (2025-11-05 07:23:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.123708518s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.765286911s trace: https://jaeger.dev.storj.io/trace/75afb5f91d3935bc FAIL (2025-11-05 07:23:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== v 5 07:09:30.093538136 (139h29m1.789386889s) [upload=122911]: Uploading segments... Nov 5 07:09:30.093576248 (139h29m1.789424992s) [upload=122911]: Got next segment. Inline: false Nov 5 07:09:30.093604072 (139h29m1.789452825s) [upload=122911]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 07:09:30.093608600 (139h29m1.789457343s) [upload=122911]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:30.339289624 (139h29m2.035138367s) [seg_pos={0 0}, upload=122911]: Begin upload segment... Nov 5 07:09:30.339349428 (139h29m2.035198181s) [seg_pos={0 0}, upload=122911]: Done begin upload segment. Nov 5 07:09:30.339386408 (139h29m2.035235151s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340339152 (139h29m2.036187905s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340990803 (139h29m2.036839556s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341444226 (139h29m2.03729298s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341919592 (139h29m2.037768375s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342371553 (139h29m2.038220306s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342835706 (139h29m2.038684449s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343494230 (139h29m2.039342973s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343970537 (139h29m2.03981929s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.345438873 (139h29m2.041287626s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.347000497 (139h29m2.04284925s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.348576918 (139h29m2.044425661s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.350325538 (139h29m2.046174281s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.351367712 (139h29m2.047216465s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.354636787 (139h29m2.050485571s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.356330162 (139h29m2.052178915s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357738744 (139h29m2.053587487s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357956329 (139h29m2.053805172s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:31.727744519 (139h29m3.423593262s) [upload=122911]: Next returned nil segment Nov 5 07:09:31.727759807 (139h29m3.42360856s) [upload=122911]: Waiting for error group managing segments... Nov 5 07:09:32.079494147 (139h29m3.77534289s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.084663190 (139h29m3.780511933s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.140346892 (139h29m3.836195645s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.141263718 (139h29m3.837112472s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.325083005 (139h29m4.020931758s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.327962548 (139h29m4.023811301s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.375450033 (139h29m4.071298786s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.421783842 (139h29m4.117632595s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.435408857 (139h29m4.13125761s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.450128345 (139h29m4.145977088s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.464432925 (139h29m4.160281678s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.483329487 (139h29m4.17917824s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.594589918 (139h29m4.290438661s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:10:29.406200389 (139h30m1.102049142s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:145.79.227.13:4820): protocol: rpc: dial tcp 145.79.227.13:4820: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406293496 (139h30m1.102142249s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb, address:145.79.227.16:4920): protocol: rpc: dial tcp 145.79.227.16:4920: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406452028 (139h30m1.102300781s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:145.79.227.20:4440): protocol: rpc: dial tcp 145.79.227.20:4440: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406646058 (139h30m1.102494801s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:145.79.227.39:4047): protocol: rpc: dial tcp 145.79.227.39:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406963362 (139h30m1.102812125s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: rpc: dial tcp 145.79.227.2:4131: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.407007867 (139h30m1.10285662s) [seg_pos={0 0}, upload=122911]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 5 07:10:29.648039207 (139h30m1.34388797s) [upload=122911]: Done uploading segments... Nov 5 07:10:29.648083852 (139h30m1.343932595s) [upload=122911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.765374828s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................XX................] duration: 5.820347347s trace: https://jaeger.dev.storj.io/trace/fe05d2f05e7af5f FAIL (2025-11-05 07:23:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 164056 (139h28m44.854012819s) [upload=122910]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:13.414175591 (139h28m45.110024344s) [seg_pos={0 0}, upload=122910]: Begin upload segment... Nov 5 07:09:13.414231787 (139h28m45.110080541s) [seg_pos={0 0}, upload=122910]: Done begin upload segment. Nov 5 07:09:13.414274930 (139h28m45.110123683s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415327825 (139h28m45.111176578s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415861491 (139h28m45.111710244s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416375510 (139h28m45.112224263s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416678197 (139h28m45.11252695s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417009026 (139h28m45.112857779s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417341269 (139h28m45.113190022s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417656319 (139h28m45.113505072s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.418114902 (139h28m45.113963645s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.420954530 (139h28m45.116803273s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.422854929 (139h28m45.118703682s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.424915342 (139h28m45.120764095s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.426819858 (139h28m45.122668611s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.428917523 (139h28m45.124766276s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.430615777 (139h28m45.12646452s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.432400986 (139h28m45.128249739s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:15.183539119 (139h28m46.879387872s) [upload=122910]: Next returned nil segment Nov 5 07:09:15.183553485 (139h28m46.879402239s) [upload=122910]: Waiting for error group managing segments... Nov 5 07:09:15.515060579 (139h28m47.210909332s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.562983674 (139h28m47.258832417s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.584642687 (139h28m47.28049144s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.591557684 (139h28m47.287406427s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.601517589 (139h28m47.297366333s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.688637153 (139h28m47.384485906s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:42.448516803 (139h29m14.144365566s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.448842534 (139h29m14.144691287s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: rpc: dial tcp 145.79.227.13:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449324783 (139h29m14.145173536s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449692342 (139h29m14.145541085s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450039493 (139h29m14.145888236s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450238873 (139h29m14.146087626s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450426109 (139h29m14.146274862s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450710671 (139h29m14.146559434s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: rpc: dial tcp 145.79.227.20:4437: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450894401 (139h29m14.146743154s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: rpc: dial tcp 145.79.227.25:4617: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.451068282 (139h29m14.146917036s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.453469054 (139h29m14.149317797s) [seg_pos={0 0}, upload=122910]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:43.162545170 (139h29m14.858393923s) [upload=122910]: Done uploading segments... Nov 5 07:09:43.162601467 (139h29m14.85845022s) [upload=122910]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 5.820375401s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.234084119s trace: https://jaeger.dev.storj.io/trace/1b48011ff2f3bc78 FAIL (2025-11-05 07:22:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.234117492s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................XX...............] duration: 5.77482363s trace: https://jaeger.dev.storj.io/trace/fe05d2f05e7af5f FAIL (2025-11-05 07:22:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 164056 (139h28m44.854012819s) [upload=122910]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:13.414175591 (139h28m45.110024344s) [seg_pos={0 0}, upload=122910]: Begin upload segment... Nov 5 07:09:13.414231787 (139h28m45.110080541s) [seg_pos={0 0}, upload=122910]: Done begin upload segment. Nov 5 07:09:13.414274930 (139h28m45.110123683s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415327825 (139h28m45.111176578s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415861491 (139h28m45.111710244s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416375510 (139h28m45.112224263s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416678197 (139h28m45.11252695s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417009026 (139h28m45.112857779s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417341269 (139h28m45.113190022s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417656319 (139h28m45.113505072s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.418114902 (139h28m45.113963645s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.420954530 (139h28m45.116803273s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.422854929 (139h28m45.118703682s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.424915342 (139h28m45.120764095s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.426819858 (139h28m45.122668611s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.428917523 (139h28m45.124766276s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.430615777 (139h28m45.12646452s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.432400986 (139h28m45.128249739s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:15.183539119 (139h28m46.879387872s) [upload=122910]: Next returned nil segment Nov 5 07:09:15.183553485 (139h28m46.879402239s) [upload=122910]: Waiting for error group managing segments... Nov 5 07:09:15.515060579 (139h28m47.210909332s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.562983674 (139h28m47.258832417s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.584642687 (139h28m47.28049144s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.591557684 (139h28m47.287406427s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.601517589 (139h28m47.297366333s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.688637153 (139h28m47.384485906s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:42.448516803 (139h29m14.144365566s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.448842534 (139h29m14.144691287s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: rpc: dial tcp 145.79.227.13:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449324783 (139h29m14.145173536s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449692342 (139h29m14.145541085s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450039493 (139h29m14.145888236s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450238873 (139h29m14.146087626s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450426109 (139h29m14.146274862s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450710671 (139h29m14.146559434s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: rpc: dial tcp 145.79.227.20:4437: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450894401 (139h29m14.146743154s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: rpc: dial tcp 145.79.227.25:4617: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.451068282 (139h29m14.146917036s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.453469054 (139h29m14.149317797s) [seg_pos={0 0}, upload=122910]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:43.162545170 (139h29m14.858393923s) [upload=122910]: Done uploading segments... Nov 5 07:09:43.162601467 (139h29m14.85845022s) [upload=122910]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 5.774852545s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.99481879s trace: https://jaeger.dev.storj.io/trace/75afb5f91d3935bc FAIL (2025-11-05 07:22:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== v 5 07:09:30.093538136 (139h29m1.789386889s) [upload=122911]: Uploading segments... Nov 5 07:09:30.093576248 (139h29m1.789424992s) [upload=122911]: Got next segment. Inline: false Nov 5 07:09:30.093604072 (139h29m1.789452825s) [upload=122911]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 07:09:30.093608600 (139h29m1.789457343s) [upload=122911]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:30.339289624 (139h29m2.035138367s) [seg_pos={0 0}, upload=122911]: Begin upload segment... Nov 5 07:09:30.339349428 (139h29m2.035198181s) [seg_pos={0 0}, upload=122911]: Done begin upload segment. Nov 5 07:09:30.339386408 (139h29m2.035235151s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340339152 (139h29m2.036187905s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340990803 (139h29m2.036839556s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341444226 (139h29m2.03729298s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341919592 (139h29m2.037768375s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342371553 (139h29m2.038220306s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342835706 (139h29m2.038684449s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343494230 (139h29m2.039342973s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343970537 (139h29m2.03981929s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.345438873 (139h29m2.041287626s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.347000497 (139h29m2.04284925s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.348576918 (139h29m2.044425661s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.350325538 (139h29m2.046174281s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.351367712 (139h29m2.047216465s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.354636787 (139h29m2.050485571s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.356330162 (139h29m2.052178915s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357738744 (139h29m2.053587487s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357956329 (139h29m2.053805172s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:31.727744519 (139h29m3.423593262s) [upload=122911]: Next returned nil segment Nov 5 07:09:31.727759807 (139h29m3.42360856s) [upload=122911]: Waiting for error group managing segments... Nov 5 07:09:32.079494147 (139h29m3.77534289s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.084663190 (139h29m3.780511933s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.140346892 (139h29m3.836195645s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.141263718 (139h29m3.837112472s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.325083005 (139h29m4.020931758s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.327962548 (139h29m4.023811301s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.375450033 (139h29m4.071298786s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.421783842 (139h29m4.117632595s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.435408857 (139h29m4.13125761s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.450128345 (139h29m4.145977088s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.464432925 (139h29m4.160281678s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.483329487 (139h29m4.17917824s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.594589918 (139h29m4.290438661s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:10:29.406200389 (139h30m1.102049142s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:145.79.227.13:4820): protocol: rpc: dial tcp 145.79.227.13:4820: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406293496 (139h30m1.102142249s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb, address:145.79.227.16:4920): protocol: rpc: dial tcp 145.79.227.16:4920: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406452028 (139h30m1.102300781s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:145.79.227.20:4440): protocol: rpc: dial tcp 145.79.227.20:4440: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406646058 (139h30m1.102494801s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:145.79.227.39:4047): protocol: rpc: dial tcp 145.79.227.39:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406963362 (139h30m1.102812125s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: rpc: dial tcp 145.79.227.2:4131: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.407007867 (139h30m1.10285662s) [seg_pos={0 0}, upload=122911]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 5 07:10:29.648039207 (139h30m1.34388797s) [upload=122911]: Done uploading segments... Nov 5 07:10:29.648083852 (139h30m1.343932595s) [upload=122911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.994854589s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 6.351242539s trace: https://jaeger.dev.storj.io/trace/1b48011ff2f3bc78 FAIL (2025-11-05 07:22:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.351306761s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................XX..............] duration: 5.809624369s trace: https://jaeger.dev.storj.io/trace/fe05d2f05e7af5f FAIL (2025-11-05 07:21:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 164056 (139h28m44.854012819s) [upload=122910]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:13.414175591 (139h28m45.110024344s) [seg_pos={0 0}, upload=122910]: Begin upload segment... Nov 5 07:09:13.414231787 (139h28m45.110080541s) [seg_pos={0 0}, upload=122910]: Done begin upload segment. Nov 5 07:09:13.414274930 (139h28m45.110123683s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415327825 (139h28m45.111176578s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415861491 (139h28m45.111710244s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416375510 (139h28m45.112224263s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416678197 (139h28m45.11252695s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417009026 (139h28m45.112857779s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417341269 (139h28m45.113190022s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417656319 (139h28m45.113505072s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.418114902 (139h28m45.113963645s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.420954530 (139h28m45.116803273s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.422854929 (139h28m45.118703682s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.424915342 (139h28m45.120764095s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.426819858 (139h28m45.122668611s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.428917523 (139h28m45.124766276s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.430615777 (139h28m45.12646452s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.432400986 (139h28m45.128249739s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:15.183539119 (139h28m46.879387872s) [upload=122910]: Next returned nil segment Nov 5 07:09:15.183553485 (139h28m46.879402239s) [upload=122910]: Waiting for error group managing segments... Nov 5 07:09:15.515060579 (139h28m47.210909332s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.562983674 (139h28m47.258832417s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.584642687 (139h28m47.28049144s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.591557684 (139h28m47.287406427s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.601517589 (139h28m47.297366333s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.688637153 (139h28m47.384485906s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:42.448516803 (139h29m14.144365566s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.448842534 (139h29m14.144691287s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: rpc: dial tcp 145.79.227.13:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449324783 (139h29m14.145173536s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449692342 (139h29m14.145541085s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450039493 (139h29m14.145888236s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450238873 (139h29m14.146087626s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450426109 (139h29m14.146274862s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450710671 (139h29m14.146559434s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: rpc: dial tcp 145.79.227.20:4437: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450894401 (139h29m14.146743154s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: rpc: dial tcp 145.79.227.25:4617: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.451068282 (139h29m14.146917036s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.453469054 (139h29m14.149317797s) [seg_pos={0 0}, upload=122910]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:43.162545170 (139h29m14.858393923s) [upload=122910]: Done uploading segments... Nov 5 07:09:43.162601467 (139h29m14.85845022s) [upload=122910]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 5.809658806s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.816807257s trace: https://jaeger.dev.storj.io/trace/75afb5f91d3935bc FAIL (2025-11-05 07:21:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== v 5 07:09:30.093538136 (139h29m1.789386889s) [upload=122911]: Uploading segments... Nov 5 07:09:30.093576248 (139h29m1.789424992s) [upload=122911]: Got next segment. Inline: false Nov 5 07:09:30.093604072 (139h29m1.789452825s) [upload=122911]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 07:09:30.093608600 (139h29m1.789457343s) [upload=122911]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:30.339289624 (139h29m2.035138367s) [seg_pos={0 0}, upload=122911]: Begin upload segment... Nov 5 07:09:30.339349428 (139h29m2.035198181s) [seg_pos={0 0}, upload=122911]: Done begin upload segment. Nov 5 07:09:30.339386408 (139h29m2.035235151s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340339152 (139h29m2.036187905s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340990803 (139h29m2.036839556s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341444226 (139h29m2.03729298s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341919592 (139h29m2.037768375s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342371553 (139h29m2.038220306s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342835706 (139h29m2.038684449s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343494230 (139h29m2.039342973s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343970537 (139h29m2.03981929s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.345438873 (139h29m2.041287626s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.347000497 (139h29m2.04284925s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.348576918 (139h29m2.044425661s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.350325538 (139h29m2.046174281s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.351367712 (139h29m2.047216465s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.354636787 (139h29m2.050485571s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.356330162 (139h29m2.052178915s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357738744 (139h29m2.053587487s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357956329 (139h29m2.053805172s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:31.727744519 (139h29m3.423593262s) [upload=122911]: Next returned nil segment Nov 5 07:09:31.727759807 (139h29m3.42360856s) [upload=122911]: Waiting for error group managing segments... Nov 5 07:09:32.079494147 (139h29m3.77534289s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.084663190 (139h29m3.780511933s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.140346892 (139h29m3.836195645s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.141263718 (139h29m3.837112472s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.325083005 (139h29m4.020931758s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.327962548 (139h29m4.023811301s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.375450033 (139h29m4.071298786s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.421783842 (139h29m4.117632595s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.435408857 (139h29m4.13125761s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.450128345 (139h29m4.145977088s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.464432925 (139h29m4.160281678s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.483329487 (139h29m4.17917824s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.594589918 (139h29m4.290438661s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:10:29.406200389 (139h30m1.102049142s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:145.79.227.13:4820): protocol: rpc: dial tcp 145.79.227.13:4820: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406293496 (139h30m1.102142249s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb, address:145.79.227.16:4920): protocol: rpc: dial tcp 145.79.227.16:4920: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406452028 (139h30m1.102300781s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:145.79.227.20:4440): protocol: rpc: dial tcp 145.79.227.20:4440: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406646058 (139h30m1.102494801s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:145.79.227.39:4047): protocol: rpc: dial tcp 145.79.227.39:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406963362 (139h30m1.102812125s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: rpc: dial tcp 145.79.227.2:4131: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.407007867 (139h30m1.10285662s) [seg_pos={0 0}, upload=122911]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 5 07:10:29.648039207 (139h30m1.34388797s) [upload=122911]: Done uploading segments... Nov 5 07:10:29.648083852 (139h30m1.343932595s) [upload=122911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.816833376s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.463574931s trace: https://jaeger.dev.storj.io/trace/1b48011ff2f3bc78 FAIL (2025-11-05 07:20:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.463602495s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.836663728s trace: https://jaeger.dev.storj.io/trace/75afb5f91d3935bc FAIL (2025-11-05 07:20:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== v 5 07:09:30.093538136 (139h29m1.789386889s) [upload=122911]: Uploading segments... Nov 5 07:09:30.093576248 (139h29m1.789424992s) [upload=122911]: Got next segment. Inline: false Nov 5 07:09:30.093604072 (139h29m1.789452825s) [upload=122911]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 07:09:30.093608600 (139h29m1.789457343s) [upload=122911]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:30.339289624 (139h29m2.035138367s) [seg_pos={0 0}, upload=122911]: Begin upload segment... Nov 5 07:09:30.339349428 (139h29m2.035198181s) [seg_pos={0 0}, upload=122911]: Done begin upload segment. Nov 5 07:09:30.339386408 (139h29m2.035235151s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340339152 (139h29m2.036187905s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340990803 (139h29m2.036839556s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341444226 (139h29m2.03729298s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341919592 (139h29m2.037768375s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342371553 (139h29m2.038220306s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342835706 (139h29m2.038684449s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343494230 (139h29m2.039342973s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343970537 (139h29m2.03981929s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.345438873 (139h29m2.041287626s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.347000497 (139h29m2.04284925s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.348576918 (139h29m2.044425661s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.350325538 (139h29m2.046174281s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.351367712 (139h29m2.047216465s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.354636787 (139h29m2.050485571s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.356330162 (139h29m2.052178915s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357738744 (139h29m2.053587487s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357956329 (139h29m2.053805172s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:31.727744519 (139h29m3.423593262s) [upload=122911]: Next returned nil segment Nov 5 07:09:31.727759807 (139h29m3.42360856s) [upload=122911]: Waiting for error group managing segments... Nov 5 07:09:32.079494147 (139h29m3.77534289s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.084663190 (139h29m3.780511933s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.140346892 (139h29m3.836195645s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.141263718 (139h29m3.837112472s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.325083005 (139h29m4.020931758s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.327962548 (139h29m4.023811301s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.375450033 (139h29m4.071298786s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.421783842 (139h29m4.117632595s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.435408857 (139h29m4.13125761s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.450128345 (139h29m4.145977088s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.464432925 (139h29m4.160281678s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.483329487 (139h29m4.17917824s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.594589918 (139h29m4.290438661s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:10:29.406200389 (139h30m1.102049142s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:145.79.227.13:4820): protocol: rpc: dial tcp 145.79.227.13:4820: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406293496 (139h30m1.102142249s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb, address:145.79.227.16:4920): protocol: rpc: dial tcp 145.79.227.16:4920: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406452028 (139h30m1.102300781s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:145.79.227.20:4440): protocol: rpc: dial tcp 145.79.227.20:4440: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406646058 (139h30m1.102494801s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:145.79.227.39:4047): protocol: rpc: dial tcp 145.79.227.39:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406963362 (139h30m1.102812125s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: rpc: dial tcp 145.79.227.2:4131: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.407007867 (139h30m1.10285662s) [seg_pos={0 0}, upload=122911]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 5 07:10:29.648039207 (139h30m1.34388797s) [upload=122911]: Done uploading segments... Nov 5 07:10:29.648083852 (139h30m1.343932595s) [upload=122911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.836695167s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................XX.............] duration: 5.849702877s trace: https://jaeger.dev.storj.io/trace/fe05d2f05e7af5f FAIL (2025-11-05 07:20:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 164056 (139h28m44.854012819s) [upload=122910]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:13.414175591 (139h28m45.110024344s) [seg_pos={0 0}, upload=122910]: Begin upload segment... Nov 5 07:09:13.414231787 (139h28m45.110080541s) [seg_pos={0 0}, upload=122910]: Done begin upload segment. Nov 5 07:09:13.414274930 (139h28m45.110123683s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415327825 (139h28m45.111176578s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415861491 (139h28m45.111710244s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416375510 (139h28m45.112224263s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416678197 (139h28m45.11252695s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417009026 (139h28m45.112857779s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417341269 (139h28m45.113190022s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417656319 (139h28m45.113505072s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.418114902 (139h28m45.113963645s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.420954530 (139h28m45.116803273s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.422854929 (139h28m45.118703682s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.424915342 (139h28m45.120764095s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.426819858 (139h28m45.122668611s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.428917523 (139h28m45.124766276s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.430615777 (139h28m45.12646452s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.432400986 (139h28m45.128249739s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:15.183539119 (139h28m46.879387872s) [upload=122910]: Next returned nil segment Nov 5 07:09:15.183553485 (139h28m46.879402239s) [upload=122910]: Waiting for error group managing segments... Nov 5 07:09:15.515060579 (139h28m47.210909332s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.562983674 (139h28m47.258832417s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.584642687 (139h28m47.28049144s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.591557684 (139h28m47.287406427s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.601517589 (139h28m47.297366333s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.688637153 (139h28m47.384485906s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:42.448516803 (139h29m14.144365566s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.448842534 (139h29m14.144691287s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: rpc: dial tcp 145.79.227.13:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449324783 (139h29m14.145173536s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449692342 (139h29m14.145541085s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450039493 (139h29m14.145888236s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450238873 (139h29m14.146087626s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450426109 (139h29m14.146274862s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450710671 (139h29m14.146559434s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: rpc: dial tcp 145.79.227.20:4437: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450894401 (139h29m14.146743154s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: rpc: dial tcp 145.79.227.25:4617: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.451068282 (139h29m14.146917036s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.453469054 (139h29m14.149317797s) [seg_pos={0 0}, upload=122910]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:43.162545170 (139h29m14.858393923s) [upload=122910]: Done uploading segments... Nov 5 07:09:43.162601467 (139h29m14.85845022s) [upload=122910]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 5.849734417s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.376579748s trace: https://jaeger.dev.storj.io/trace/1b48011ff2f3bc78 FAIL (2025-11-05 07:20:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.376604806s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.74740252s trace: https://jaeger.dev.storj.io/trace/75afb5f91d3935bc FAIL (2025-11-05 07:20:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== v 5 07:09:30.093538136 (139h29m1.789386889s) [upload=122911]: Uploading segments... Nov 5 07:09:30.093576248 (139h29m1.789424992s) [upload=122911]: Got next segment. Inline: false Nov 5 07:09:30.093604072 (139h29m1.789452825s) [upload=122911]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 07:09:30.093608600 (139h29m1.789457343s) [upload=122911]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:30.339289624 (139h29m2.035138367s) [seg_pos={0 0}, upload=122911]: Begin upload segment... Nov 5 07:09:30.339349428 (139h29m2.035198181s) [seg_pos={0 0}, upload=122911]: Done begin upload segment. Nov 5 07:09:30.339386408 (139h29m2.035235151s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340339152 (139h29m2.036187905s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340990803 (139h29m2.036839556s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341444226 (139h29m2.03729298s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341919592 (139h29m2.037768375s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342371553 (139h29m2.038220306s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342835706 (139h29m2.038684449s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343494230 (139h29m2.039342973s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343970537 (139h29m2.03981929s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.345438873 (139h29m2.041287626s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.347000497 (139h29m2.04284925s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.348576918 (139h29m2.044425661s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.350325538 (139h29m2.046174281s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.351367712 (139h29m2.047216465s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.354636787 (139h29m2.050485571s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.356330162 (139h29m2.052178915s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357738744 (139h29m2.053587487s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357956329 (139h29m2.053805172s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:31.727744519 (139h29m3.423593262s) [upload=122911]: Next returned nil segment Nov 5 07:09:31.727759807 (139h29m3.42360856s) [upload=122911]: Waiting for error group managing segments... Nov 5 07:09:32.079494147 (139h29m3.77534289s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.084663190 (139h29m3.780511933s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.140346892 (139h29m3.836195645s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.141263718 (139h29m3.837112472s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.325083005 (139h29m4.020931758s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.327962548 (139h29m4.023811301s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.375450033 (139h29m4.071298786s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.421783842 (139h29m4.117632595s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.435408857 (139h29m4.13125761s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.450128345 (139h29m4.145977088s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.464432925 (139h29m4.160281678s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.483329487 (139h29m4.17917824s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.594589918 (139h29m4.290438661s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:10:29.406200389 (139h30m1.102049142s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:145.79.227.13:4820): protocol: rpc: dial tcp 145.79.227.13:4820: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406293496 (139h30m1.102142249s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb, address:145.79.227.16:4920): protocol: rpc: dial tcp 145.79.227.16:4920: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406452028 (139h30m1.102300781s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:145.79.227.20:4440): protocol: rpc: dial tcp 145.79.227.20:4440: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406646058 (139h30m1.102494801s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:145.79.227.39:4047): protocol: rpc: dial tcp 145.79.227.39:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406963362 (139h30m1.102812125s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: rpc: dial tcp 145.79.227.2:4131: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.407007867 (139h30m1.10285662s) [seg_pos={0 0}, upload=122911]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 5 07:10:29.648039207 (139h30m1.34388797s) [upload=122911]: Done uploading segments... Nov 5 07:10:29.648083852 (139h30m1.343932595s) [upload=122911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.747434s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.278511665s trace: https://jaeger.dev.storj.io/trace/1b48011ff2f3bc78 FAIL (2025-11-05 07:20:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.278531965s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................XX............] duration: 5.721877392s trace: https://jaeger.dev.storj.io/trace/fe05d2f05e7af5f FAIL (2025-11-05 07:19:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 164056 (139h28m44.854012819s) [upload=122910]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:13.414175591 (139h28m45.110024344s) [seg_pos={0 0}, upload=122910]: Begin upload segment... Nov 5 07:09:13.414231787 (139h28m45.110080541s) [seg_pos={0 0}, upload=122910]: Done begin upload segment. Nov 5 07:09:13.414274930 (139h28m45.110123683s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415327825 (139h28m45.111176578s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415861491 (139h28m45.111710244s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416375510 (139h28m45.112224263s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416678197 (139h28m45.11252695s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417009026 (139h28m45.112857779s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417341269 (139h28m45.113190022s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417656319 (139h28m45.113505072s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.418114902 (139h28m45.113963645s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.420954530 (139h28m45.116803273s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.422854929 (139h28m45.118703682s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.424915342 (139h28m45.120764095s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.426819858 (139h28m45.122668611s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.428917523 (139h28m45.124766276s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.430615777 (139h28m45.12646452s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.432400986 (139h28m45.128249739s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:15.183539119 (139h28m46.879387872s) [upload=122910]: Next returned nil segment Nov 5 07:09:15.183553485 (139h28m46.879402239s) [upload=122910]: Waiting for error group managing segments... Nov 5 07:09:15.515060579 (139h28m47.210909332s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.562983674 (139h28m47.258832417s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.584642687 (139h28m47.28049144s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.591557684 (139h28m47.287406427s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.601517589 (139h28m47.297366333s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.688637153 (139h28m47.384485906s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:42.448516803 (139h29m14.144365566s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.448842534 (139h29m14.144691287s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: rpc: dial tcp 145.79.227.13:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449324783 (139h29m14.145173536s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449692342 (139h29m14.145541085s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450039493 (139h29m14.145888236s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450238873 (139h29m14.146087626s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450426109 (139h29m14.146274862s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450710671 (139h29m14.146559434s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: rpc: dial tcp 145.79.227.20:4437: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450894401 (139h29m14.146743154s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: rpc: dial tcp 145.79.227.25:4617: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.451068282 (139h29m14.146917036s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.453469054 (139h29m14.149317797s) [seg_pos={0 0}, upload=122910]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:43.162545170 (139h29m14.858393923s) [upload=122910]: Done uploading segments... Nov 5 07:09:43.162601467 (139h29m14.85845022s) [upload=122910]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 5.721925542s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.28456387s trace: https://jaeger.dev.storj.io/trace/1b48011ff2f3bc78 FAIL (2025-11-05 07:19:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.284590019s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.223430117s trace: https://jaeger.dev.storj.io/trace/75afb5f91d3935bc FAIL (2025-11-05 07:19:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== v 5 07:09:30.093538136 (139h29m1.789386889s) [upload=122911]: Uploading segments... Nov 5 07:09:30.093576248 (139h29m1.789424992s) [upload=122911]: Got next segment. Inline: false Nov 5 07:09:30.093604072 (139h29m1.789452825s) [upload=122911]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 07:09:30.093608600 (139h29m1.789457343s) [upload=122911]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:30.339289624 (139h29m2.035138367s) [seg_pos={0 0}, upload=122911]: Begin upload segment... Nov 5 07:09:30.339349428 (139h29m2.035198181s) [seg_pos={0 0}, upload=122911]: Done begin upload segment. Nov 5 07:09:30.339386408 (139h29m2.035235151s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340339152 (139h29m2.036187905s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340990803 (139h29m2.036839556s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341444226 (139h29m2.03729298s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341919592 (139h29m2.037768375s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342371553 (139h29m2.038220306s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342835706 (139h29m2.038684449s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343494230 (139h29m2.039342973s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343970537 (139h29m2.03981929s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.345438873 (139h29m2.041287626s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.347000497 (139h29m2.04284925s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.348576918 (139h29m2.044425661s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.350325538 (139h29m2.046174281s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.351367712 (139h29m2.047216465s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.354636787 (139h29m2.050485571s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.356330162 (139h29m2.052178915s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357738744 (139h29m2.053587487s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357956329 (139h29m2.053805172s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:31.727744519 (139h29m3.423593262s) [upload=122911]: Next returned nil segment Nov 5 07:09:31.727759807 (139h29m3.42360856s) [upload=122911]: Waiting for error group managing segments... Nov 5 07:09:32.079494147 (139h29m3.77534289s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.084663190 (139h29m3.780511933s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.140346892 (139h29m3.836195645s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.141263718 (139h29m3.837112472s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.325083005 (139h29m4.020931758s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.327962548 (139h29m4.023811301s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.375450033 (139h29m4.071298786s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.421783842 (139h29m4.117632595s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.435408857 (139h29m4.13125761s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.450128345 (139h29m4.145977088s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.464432925 (139h29m4.160281678s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.483329487 (139h29m4.17917824s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.594589918 (139h29m4.290438661s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:10:29.406200389 (139h30m1.102049142s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:145.79.227.13:4820): protocol: rpc: dial tcp 145.79.227.13:4820: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406293496 (139h30m1.102142249s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb, address:145.79.227.16:4920): protocol: rpc: dial tcp 145.79.227.16:4920: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406452028 (139h30m1.102300781s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:145.79.227.20:4440): protocol: rpc: dial tcp 145.79.227.20:4440: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406646058 (139h30m1.102494801s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:145.79.227.39:4047): protocol: rpc: dial tcp 145.79.227.39:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406963362 (139h30m1.102812125s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: rpc: dial tcp 145.79.227.2:4131: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.407007867 (139h30m1.10285662s) [seg_pos={0 0}, upload=122911]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 5 07:10:29.648039207 (139h30m1.34388797s) [upload=122911]: Done uploading segments... Nov 5 07:10:29.648083852 (139h30m1.343932595s) [upload=122911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.223467589s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.184032856s trace: https://jaeger.dev.storj.io/trace/1b48011ff2f3bc78 FAIL (2025-11-05 07:19:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.184059896s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.803051773s trace: https://jaeger.dev.storj.io/trace/75afb5f91d3935bc FAIL (2025-11-05 07:18:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== v 5 07:09:30.093538136 (139h29m1.789386889s) [upload=122911]: Uploading segments... Nov 5 07:09:30.093576248 (139h29m1.789424992s) [upload=122911]: Got next segment. Inline: false Nov 5 07:09:30.093604072 (139h29m1.789452825s) [upload=122911]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 07:09:30.093608600 (139h29m1.789457343s) [upload=122911]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:30.339289624 (139h29m2.035138367s) [seg_pos={0 0}, upload=122911]: Begin upload segment... Nov 5 07:09:30.339349428 (139h29m2.035198181s) [seg_pos={0 0}, upload=122911]: Done begin upload segment. Nov 5 07:09:30.339386408 (139h29m2.035235151s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340339152 (139h29m2.036187905s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340990803 (139h29m2.036839556s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341444226 (139h29m2.03729298s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341919592 (139h29m2.037768375s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342371553 (139h29m2.038220306s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342835706 (139h29m2.038684449s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343494230 (139h29m2.039342973s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343970537 (139h29m2.03981929s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.345438873 (139h29m2.041287626s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.347000497 (139h29m2.04284925s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.348576918 (139h29m2.044425661s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.350325538 (139h29m2.046174281s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.351367712 (139h29m2.047216465s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.354636787 (139h29m2.050485571s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.356330162 (139h29m2.052178915s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357738744 (139h29m2.053587487s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357956329 (139h29m2.053805172s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:31.727744519 (139h29m3.423593262s) [upload=122911]: Next returned nil segment Nov 5 07:09:31.727759807 (139h29m3.42360856s) [upload=122911]: Waiting for error group managing segments... Nov 5 07:09:32.079494147 (139h29m3.77534289s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.084663190 (139h29m3.780511933s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.140346892 (139h29m3.836195645s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.141263718 (139h29m3.837112472s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.325083005 (139h29m4.020931758s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.327962548 (139h29m4.023811301s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.375450033 (139h29m4.071298786s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.421783842 (139h29m4.117632595s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.435408857 (139h29m4.13125761s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.450128345 (139h29m4.145977088s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.464432925 (139h29m4.160281678s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.483329487 (139h29m4.17917824s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.594589918 (139h29m4.290438661s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:10:29.406200389 (139h30m1.102049142s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:145.79.227.13:4820): protocol: rpc: dial tcp 145.79.227.13:4820: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406293496 (139h30m1.102142249s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb, address:145.79.227.16:4920): protocol: rpc: dial tcp 145.79.227.16:4920: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406452028 (139h30m1.102300781s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:145.79.227.20:4440): protocol: rpc: dial tcp 145.79.227.20:4440: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406646058 (139h30m1.102494801s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:145.79.227.39:4047): protocol: rpc: dial tcp 145.79.227.39:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406963362 (139h30m1.102812125s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: rpc: dial tcp 145.79.227.2:4131: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.407007867 (139h30m1.10285662s) [seg_pos={0 0}, upload=122911]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 5 07:10:29.648039207 (139h30m1.34388797s) [upload=122911]: Done uploading segments... Nov 5 07:10:29.648083852 (139h30m1.343932595s) [upload=122911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.803122858s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................XX...........] duration: 5.754499379s trace: https://jaeger.dev.storj.io/trace/fe05d2f05e7af5f FAIL (2025-11-05 07:18:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 164056 (139h28m44.854012819s) [upload=122910]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:13.414175591 (139h28m45.110024344s) [seg_pos={0 0}, upload=122910]: Begin upload segment... Nov 5 07:09:13.414231787 (139h28m45.110080541s) [seg_pos={0 0}, upload=122910]: Done begin upload segment. Nov 5 07:09:13.414274930 (139h28m45.110123683s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415327825 (139h28m45.111176578s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415861491 (139h28m45.111710244s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416375510 (139h28m45.112224263s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416678197 (139h28m45.11252695s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417009026 (139h28m45.112857779s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417341269 (139h28m45.113190022s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417656319 (139h28m45.113505072s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.418114902 (139h28m45.113963645s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.420954530 (139h28m45.116803273s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.422854929 (139h28m45.118703682s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.424915342 (139h28m45.120764095s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.426819858 (139h28m45.122668611s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.428917523 (139h28m45.124766276s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.430615777 (139h28m45.12646452s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.432400986 (139h28m45.128249739s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:15.183539119 (139h28m46.879387872s) [upload=122910]: Next returned nil segment Nov 5 07:09:15.183553485 (139h28m46.879402239s) [upload=122910]: Waiting for error group managing segments... Nov 5 07:09:15.515060579 (139h28m47.210909332s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.562983674 (139h28m47.258832417s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.584642687 (139h28m47.28049144s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.591557684 (139h28m47.287406427s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.601517589 (139h28m47.297366333s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.688637153 (139h28m47.384485906s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:42.448516803 (139h29m14.144365566s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.448842534 (139h29m14.144691287s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: rpc: dial tcp 145.79.227.13:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449324783 (139h29m14.145173536s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449692342 (139h29m14.145541085s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450039493 (139h29m14.145888236s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450238873 (139h29m14.146087626s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450426109 (139h29m14.146274862s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450710671 (139h29m14.146559434s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: rpc: dial tcp 145.79.227.20:4437: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450894401 (139h29m14.146743154s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: rpc: dial tcp 145.79.227.25:4617: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.451068282 (139h29m14.146917036s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.453469054 (139h29m14.149317797s) [seg_pos={0 0}, upload=122910]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:43.162545170 (139h29m14.858393923s) [upload=122910]: Done uploading segments... Nov 5 07:09:43.162601467 (139h29m14.85845022s) [upload=122910]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 5.754546109s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.761353824s trace: https://jaeger.dev.storj.io/trace/75afb5f91d3935bc FAIL (2025-11-05 07:18:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== v 5 07:09:30.093538136 (139h29m1.789386889s) [upload=122911]: Uploading segments... Nov 5 07:09:30.093576248 (139h29m1.789424992s) [upload=122911]: Got next segment. Inline: false Nov 5 07:09:30.093604072 (139h29m1.789452825s) [upload=122911]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 07:09:30.093608600 (139h29m1.789457343s) [upload=122911]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:30.339289624 (139h29m2.035138367s) [seg_pos={0 0}, upload=122911]: Begin upload segment... Nov 5 07:09:30.339349428 (139h29m2.035198181s) [seg_pos={0 0}, upload=122911]: Done begin upload segment. Nov 5 07:09:30.339386408 (139h29m2.035235151s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340339152 (139h29m2.036187905s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340990803 (139h29m2.036839556s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341444226 (139h29m2.03729298s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341919592 (139h29m2.037768375s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342371553 (139h29m2.038220306s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342835706 (139h29m2.038684449s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343494230 (139h29m2.039342973s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343970537 (139h29m2.03981929s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.345438873 (139h29m2.041287626s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.347000497 (139h29m2.04284925s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.348576918 (139h29m2.044425661s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.350325538 (139h29m2.046174281s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.351367712 (139h29m2.047216465s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.354636787 (139h29m2.050485571s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.356330162 (139h29m2.052178915s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357738744 (139h29m2.053587487s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357956329 (139h29m2.053805172s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:31.727744519 (139h29m3.423593262s) [upload=122911]: Next returned nil segment Nov 5 07:09:31.727759807 (139h29m3.42360856s) [upload=122911]: Waiting for error group managing segments... Nov 5 07:09:32.079494147 (139h29m3.77534289s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.084663190 (139h29m3.780511933s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.140346892 (139h29m3.836195645s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.141263718 (139h29m3.837112472s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.325083005 (139h29m4.020931758s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.327962548 (139h29m4.023811301s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.375450033 (139h29m4.071298786s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.421783842 (139h29m4.117632595s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.435408857 (139h29m4.13125761s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.450128345 (139h29m4.145977088s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.464432925 (139h29m4.160281678s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.483329487 (139h29m4.17917824s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.594589918 (139h29m4.290438661s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:10:29.406200389 (139h30m1.102049142s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:145.79.227.13:4820): protocol: rpc: dial tcp 145.79.227.13:4820: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406293496 (139h30m1.102142249s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb, address:145.79.227.16:4920): protocol: rpc: dial tcp 145.79.227.16:4920: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406452028 (139h30m1.102300781s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:145.79.227.20:4440): protocol: rpc: dial tcp 145.79.227.20:4440: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406646058 (139h30m1.102494801s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:145.79.227.39:4047): protocol: rpc: dial tcp 145.79.227.39:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406963362 (139h30m1.102812125s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: rpc: dial tcp 145.79.227.2:4131: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.407007867 (139h30m1.10285662s) [seg_pos={0 0}, upload=122911]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 5 07:10:29.648039207 (139h30m1.34388797s) [upload=122911]: Done uploading segments... Nov 5 07:10:29.648083852 (139h30m1.343932595s) [upload=122911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.761383771s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.578531485s trace: https://jaeger.dev.storj.io/trace/1b48011ff2f3bc78 FAIL (2025-11-05 07:18:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.578559018s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................XX..........] duration: 5.638462718s trace: https://jaeger.dev.storj.io/trace/fe05d2f05e7af5f FAIL (2025-11-05 07:17:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 164056 (139h28m44.854012819s) [upload=122910]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:13.414175591 (139h28m45.110024344s) [seg_pos={0 0}, upload=122910]: Begin upload segment... Nov 5 07:09:13.414231787 (139h28m45.110080541s) [seg_pos={0 0}, upload=122910]: Done begin upload segment. Nov 5 07:09:13.414274930 (139h28m45.110123683s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415327825 (139h28m45.111176578s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415861491 (139h28m45.111710244s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416375510 (139h28m45.112224263s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416678197 (139h28m45.11252695s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417009026 (139h28m45.112857779s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417341269 (139h28m45.113190022s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417656319 (139h28m45.113505072s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.418114902 (139h28m45.113963645s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.420954530 (139h28m45.116803273s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.422854929 (139h28m45.118703682s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.424915342 (139h28m45.120764095s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.426819858 (139h28m45.122668611s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.428917523 (139h28m45.124766276s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.430615777 (139h28m45.12646452s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.432400986 (139h28m45.128249739s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:15.183539119 (139h28m46.879387872s) [upload=122910]: Next returned nil segment Nov 5 07:09:15.183553485 (139h28m46.879402239s) [upload=122910]: Waiting for error group managing segments... Nov 5 07:09:15.515060579 (139h28m47.210909332s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.562983674 (139h28m47.258832417s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.584642687 (139h28m47.28049144s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.591557684 (139h28m47.287406427s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.601517589 (139h28m47.297366333s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.688637153 (139h28m47.384485906s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:42.448516803 (139h29m14.144365566s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.448842534 (139h29m14.144691287s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: rpc: dial tcp 145.79.227.13:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449324783 (139h29m14.145173536s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449692342 (139h29m14.145541085s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450039493 (139h29m14.145888236s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450238873 (139h29m14.146087626s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450426109 (139h29m14.146274862s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450710671 (139h29m14.146559434s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: rpc: dial tcp 145.79.227.20:4437: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450894401 (139h29m14.146743154s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: rpc: dial tcp 145.79.227.25:4617: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.451068282 (139h29m14.146917036s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.453469054 (139h29m14.149317797s) [seg_pos={0 0}, upload=122910]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:43.162545170 (139h29m14.858393923s) [upload=122910]: Done uploading segments... Nov 5 07:09:43.162601467 (139h29m14.85845022s) [upload=122910]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 5.63849598s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.567645443s trace: https://jaeger.dev.storj.io/trace/1b48011ff2f3bc78 FAIL (2025-11-05 07:17:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.567672905s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................XX.........] duration: 6.054691666s trace: https://jaeger.dev.storj.io/trace/fe05d2f05e7af5f FAIL (2025-11-05 07:17:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 164056 (139h28m44.854012819s) [upload=122910]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:13.414175591 (139h28m45.110024344s) [seg_pos={0 0}, upload=122910]: Begin upload segment... Nov 5 07:09:13.414231787 (139h28m45.110080541s) [seg_pos={0 0}, upload=122910]: Done begin upload segment. Nov 5 07:09:13.414274930 (139h28m45.110123683s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415327825 (139h28m45.111176578s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415861491 (139h28m45.111710244s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416375510 (139h28m45.112224263s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416678197 (139h28m45.11252695s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417009026 (139h28m45.112857779s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417341269 (139h28m45.113190022s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417656319 (139h28m45.113505072s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.418114902 (139h28m45.113963645s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.420954530 (139h28m45.116803273s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.422854929 (139h28m45.118703682s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.424915342 (139h28m45.120764095s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.426819858 (139h28m45.122668611s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.428917523 (139h28m45.124766276s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.430615777 (139h28m45.12646452s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.432400986 (139h28m45.128249739s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:15.183539119 (139h28m46.879387872s) [upload=122910]: Next returned nil segment Nov 5 07:09:15.183553485 (139h28m46.879402239s) [upload=122910]: Waiting for error group managing segments... Nov 5 07:09:15.515060579 (139h28m47.210909332s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.562983674 (139h28m47.258832417s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.584642687 (139h28m47.28049144s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.591557684 (139h28m47.287406427s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.601517589 (139h28m47.297366333s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.688637153 (139h28m47.384485906s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:42.448516803 (139h29m14.144365566s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.448842534 (139h29m14.144691287s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: rpc: dial tcp 145.79.227.13:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449324783 (139h29m14.145173536s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449692342 (139h29m14.145541085s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450039493 (139h29m14.145888236s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450238873 (139h29m14.146087626s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450426109 (139h29m14.146274862s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450710671 (139h29m14.146559434s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: rpc: dial tcp 145.79.227.20:4437: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450894401 (139h29m14.146743154s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: rpc: dial tcp 145.79.227.25:4617: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.451068282 (139h29m14.146917036s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.453469054 (139h29m14.149317797s) [seg_pos={0 0}, upload=122910]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:43.162545170 (139h29m14.858393923s) [upload=122910]: Done uploading segments... Nov 5 07:09:43.162601467 (139h29m14.85845022s) [upload=122910]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 6.054723738s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.659952052s trace: https://jaeger.dev.storj.io/trace/75afb5f91d3935bc FAIL (2025-11-05 07:17:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== v 5 07:09:30.093538136 (139h29m1.789386889s) [upload=122911]: Uploading segments... Nov 5 07:09:30.093576248 (139h29m1.789424992s) [upload=122911]: Got next segment. Inline: false Nov 5 07:09:30.093604072 (139h29m1.789452825s) [upload=122911]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 07:09:30.093608600 (139h29m1.789457343s) [upload=122911]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:30.339289624 (139h29m2.035138367s) [seg_pos={0 0}, upload=122911]: Begin upload segment... Nov 5 07:09:30.339349428 (139h29m2.035198181s) [seg_pos={0 0}, upload=122911]: Done begin upload segment. Nov 5 07:09:30.339386408 (139h29m2.035235151s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340339152 (139h29m2.036187905s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340990803 (139h29m2.036839556s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341444226 (139h29m2.03729298s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341919592 (139h29m2.037768375s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342371553 (139h29m2.038220306s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342835706 (139h29m2.038684449s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343494230 (139h29m2.039342973s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343970537 (139h29m2.03981929s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.345438873 (139h29m2.041287626s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.347000497 (139h29m2.04284925s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.348576918 (139h29m2.044425661s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.350325538 (139h29m2.046174281s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.351367712 (139h29m2.047216465s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.354636787 (139h29m2.050485571s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.356330162 (139h29m2.052178915s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357738744 (139h29m2.053587487s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357956329 (139h29m2.053805172s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:31.727744519 (139h29m3.423593262s) [upload=122911]: Next returned nil segment Nov 5 07:09:31.727759807 (139h29m3.42360856s) [upload=122911]: Waiting for error group managing segments... Nov 5 07:09:32.079494147 (139h29m3.77534289s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.084663190 (139h29m3.780511933s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.140346892 (139h29m3.836195645s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.141263718 (139h29m3.837112472s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.325083005 (139h29m4.020931758s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.327962548 (139h29m4.023811301s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.375450033 (139h29m4.071298786s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.421783842 (139h29m4.117632595s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.435408857 (139h29m4.13125761s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.450128345 (139h29m4.145977088s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.464432925 (139h29m4.160281678s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.483329487 (139h29m4.17917824s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.594589918 (139h29m4.290438661s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:10:29.406200389 (139h30m1.102049142s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:145.79.227.13:4820): protocol: rpc: dial tcp 145.79.227.13:4820: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406293496 (139h30m1.102142249s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb, address:145.79.227.16:4920): protocol: rpc: dial tcp 145.79.227.16:4920: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406452028 (139h30m1.102300781s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:145.79.227.20:4440): protocol: rpc: dial tcp 145.79.227.20:4440: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406646058 (139h30m1.102494801s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:145.79.227.39:4047): protocol: rpc: dial tcp 145.79.227.39:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406963362 (139h30m1.102812125s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: rpc: dial tcp 145.79.227.2:4131: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.407007867 (139h30m1.10285662s) [seg_pos={0 0}, upload=122911]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 5 07:10:29.648039207 (139h30m1.34388797s) [upload=122911]: Done uploading segments... Nov 5 07:10:29.648083852 (139h30m1.343932595s) [upload=122911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.659981129s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.386568157s trace: https://jaeger.dev.storj.io/trace/1b48011ff2f3bc78 FAIL (2025-11-05 07:16:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.386593183s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................XX........] duration: 5.998706148s trace: https://jaeger.dev.storj.io/trace/fe05d2f05e7af5f FAIL (2025-11-05 07:16:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 164056 (139h28m44.854012819s) [upload=122910]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:13.414175591 (139h28m45.110024344s) [seg_pos={0 0}, upload=122910]: Begin upload segment... Nov 5 07:09:13.414231787 (139h28m45.110080541s) [seg_pos={0 0}, upload=122910]: Done begin upload segment. Nov 5 07:09:13.414274930 (139h28m45.110123683s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415327825 (139h28m45.111176578s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415861491 (139h28m45.111710244s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416375510 (139h28m45.112224263s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416678197 (139h28m45.11252695s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417009026 (139h28m45.112857779s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417341269 (139h28m45.113190022s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417656319 (139h28m45.113505072s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.418114902 (139h28m45.113963645s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.420954530 (139h28m45.116803273s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.422854929 (139h28m45.118703682s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.424915342 (139h28m45.120764095s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.426819858 (139h28m45.122668611s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.428917523 (139h28m45.124766276s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.430615777 (139h28m45.12646452s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.432400986 (139h28m45.128249739s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:15.183539119 (139h28m46.879387872s) [upload=122910]: Next returned nil segment Nov 5 07:09:15.183553485 (139h28m46.879402239s) [upload=122910]: Waiting for error group managing segments... Nov 5 07:09:15.515060579 (139h28m47.210909332s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.562983674 (139h28m47.258832417s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.584642687 (139h28m47.28049144s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.591557684 (139h28m47.287406427s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.601517589 (139h28m47.297366333s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.688637153 (139h28m47.384485906s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:42.448516803 (139h29m14.144365566s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.448842534 (139h29m14.144691287s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: rpc: dial tcp 145.79.227.13:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449324783 (139h29m14.145173536s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449692342 (139h29m14.145541085s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450039493 (139h29m14.145888236s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450238873 (139h29m14.146087626s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450426109 (139h29m14.146274862s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450710671 (139h29m14.146559434s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: rpc: dial tcp 145.79.227.20:4437: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450894401 (139h29m14.146743154s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: rpc: dial tcp 145.79.227.25:4617: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.451068282 (139h29m14.146917036s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.453469054 (139h29m14.149317797s) [seg_pos={0 0}, upload=122910]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:43.162545170 (139h29m14.858393923s) [upload=122910]: Done uploading segments... Nov 5 07:09:43.162601467 (139h29m14.85845022s) [upload=122910]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 5.998741486s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.680588858s trace: https://jaeger.dev.storj.io/trace/75afb5f91d3935bc FAIL (2025-11-05 07:16:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== v 5 07:09:30.093538136 (139h29m1.789386889s) [upload=122911]: Uploading segments... Nov 5 07:09:30.093576248 (139h29m1.789424992s) [upload=122911]: Got next segment. Inline: false Nov 5 07:09:30.093604072 (139h29m1.789452825s) [upload=122911]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 07:09:30.093608600 (139h29m1.789457343s) [upload=122911]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:30.339289624 (139h29m2.035138367s) [seg_pos={0 0}, upload=122911]: Begin upload segment... Nov 5 07:09:30.339349428 (139h29m2.035198181s) [seg_pos={0 0}, upload=122911]: Done begin upload segment. Nov 5 07:09:30.339386408 (139h29m2.035235151s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340339152 (139h29m2.036187905s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340990803 (139h29m2.036839556s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341444226 (139h29m2.03729298s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341919592 (139h29m2.037768375s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342371553 (139h29m2.038220306s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342835706 (139h29m2.038684449s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343494230 (139h29m2.039342973s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343970537 (139h29m2.03981929s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.345438873 (139h29m2.041287626s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.347000497 (139h29m2.04284925s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.348576918 (139h29m2.044425661s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.350325538 (139h29m2.046174281s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.351367712 (139h29m2.047216465s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.354636787 (139h29m2.050485571s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.356330162 (139h29m2.052178915s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357738744 (139h29m2.053587487s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357956329 (139h29m2.053805172s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:31.727744519 (139h29m3.423593262s) [upload=122911]: Next returned nil segment Nov 5 07:09:31.727759807 (139h29m3.42360856s) [upload=122911]: Waiting for error group managing segments... Nov 5 07:09:32.079494147 (139h29m3.77534289s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.084663190 (139h29m3.780511933s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.140346892 (139h29m3.836195645s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.141263718 (139h29m3.837112472s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.325083005 (139h29m4.020931758s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.327962548 (139h29m4.023811301s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.375450033 (139h29m4.071298786s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.421783842 (139h29m4.117632595s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.435408857 (139h29m4.13125761s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.450128345 (139h29m4.145977088s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.464432925 (139h29m4.160281678s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.483329487 (139h29m4.17917824s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.594589918 (139h29m4.290438661s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:10:29.406200389 (139h30m1.102049142s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:145.79.227.13:4820): protocol: rpc: dial tcp 145.79.227.13:4820: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406293496 (139h30m1.102142249s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb, address:145.79.227.16:4920): protocol: rpc: dial tcp 145.79.227.16:4920: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406452028 (139h30m1.102300781s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:145.79.227.20:4440): protocol: rpc: dial tcp 145.79.227.20:4440: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406646058 (139h30m1.102494801s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:145.79.227.39:4047): protocol: rpc: dial tcp 145.79.227.39:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406963362 (139h30m1.102812125s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: rpc: dial tcp 145.79.227.2:4131: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.407007867 (139h30m1.10285662s) [seg_pos={0 0}, upload=122911]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 5 07:10:29.648039207 (139h30m1.34388797s) [upload=122911]: Done uploading segments... Nov 5 07:10:29.648083852 (139h30m1.343932595s) [upload=122911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.680639665s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.869453429s trace: https://jaeger.dev.storj.io/trace/75afb5f91d3935bc FAIL (2025-11-05 07:16:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== v 5 07:09:30.093538136 (139h29m1.789386889s) [upload=122911]: Uploading segments... Nov 5 07:09:30.093576248 (139h29m1.789424992s) [upload=122911]: Got next segment. Inline: false Nov 5 07:09:30.093604072 (139h29m1.789452825s) [upload=122911]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 07:09:30.093608600 (139h29m1.789457343s) [upload=122911]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:30.339289624 (139h29m2.035138367s) [seg_pos={0 0}, upload=122911]: Begin upload segment... Nov 5 07:09:30.339349428 (139h29m2.035198181s) [seg_pos={0 0}, upload=122911]: Done begin upload segment. Nov 5 07:09:30.339386408 (139h29m2.035235151s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340339152 (139h29m2.036187905s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340990803 (139h29m2.036839556s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341444226 (139h29m2.03729298s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341919592 (139h29m2.037768375s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342371553 (139h29m2.038220306s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342835706 (139h29m2.038684449s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343494230 (139h29m2.039342973s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343970537 (139h29m2.03981929s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.345438873 (139h29m2.041287626s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.347000497 (139h29m2.04284925s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.348576918 (139h29m2.044425661s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.350325538 (139h29m2.046174281s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.351367712 (139h29m2.047216465s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.354636787 (139h29m2.050485571s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.356330162 (139h29m2.052178915s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357738744 (139h29m2.053587487s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357956329 (139h29m2.053805172s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:31.727744519 (139h29m3.423593262s) [upload=122911]: Next returned nil segment Nov 5 07:09:31.727759807 (139h29m3.42360856s) [upload=122911]: Waiting for error group managing segments... Nov 5 07:09:32.079494147 (139h29m3.77534289s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.084663190 (139h29m3.780511933s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.140346892 (139h29m3.836195645s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.141263718 (139h29m3.837112472s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.325083005 (139h29m4.020931758s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.327962548 (139h29m4.023811301s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.375450033 (139h29m4.071298786s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.421783842 (139h29m4.117632595s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.435408857 (139h29m4.13125761s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.450128345 (139h29m4.145977088s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.464432925 (139h29m4.160281678s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.483329487 (139h29m4.17917824s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.594589918 (139h29m4.290438661s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:10:29.406200389 (139h30m1.102049142s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:145.79.227.13:4820): protocol: rpc: dial tcp 145.79.227.13:4820: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406293496 (139h30m1.102142249s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb, address:145.79.227.16:4920): protocol: rpc: dial tcp 145.79.227.16:4920: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406452028 (139h30m1.102300781s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:145.79.227.20:4440): protocol: rpc: dial tcp 145.79.227.20:4440: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406646058 (139h30m1.102494801s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:145.79.227.39:4047): protocol: rpc: dial tcp 145.79.227.39:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406963362 (139h30m1.102812125s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: rpc: dial tcp 145.79.227.2:4131: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.407007867 (139h30m1.10285662s) [seg_pos={0 0}, upload=122911]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 5 07:10:29.648039207 (139h30m1.34388797s) [upload=122911]: Done uploading segments... Nov 5 07:10:29.648083852 (139h30m1.343932595s) [upload=122911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.869704628s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.46217297s trace: https://jaeger.dev.storj.io/trace/1b48011ff2f3bc78 FAIL (2025-11-05 07:16:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.462196364s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................XX.......] duration: 5.941800659s trace: https://jaeger.dev.storj.io/trace/fe05d2f05e7af5f FAIL (2025-11-05 07:15:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 164056 (139h28m44.854012819s) [upload=122910]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:13.414175591 (139h28m45.110024344s) [seg_pos={0 0}, upload=122910]: Begin upload segment... Nov 5 07:09:13.414231787 (139h28m45.110080541s) [seg_pos={0 0}, upload=122910]: Done begin upload segment. Nov 5 07:09:13.414274930 (139h28m45.110123683s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415327825 (139h28m45.111176578s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415861491 (139h28m45.111710244s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416375510 (139h28m45.112224263s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416678197 (139h28m45.11252695s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417009026 (139h28m45.112857779s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417341269 (139h28m45.113190022s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417656319 (139h28m45.113505072s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.418114902 (139h28m45.113963645s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.420954530 (139h28m45.116803273s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.422854929 (139h28m45.118703682s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.424915342 (139h28m45.120764095s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.426819858 (139h28m45.122668611s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.428917523 (139h28m45.124766276s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.430615777 (139h28m45.12646452s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.432400986 (139h28m45.128249739s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:15.183539119 (139h28m46.879387872s) [upload=122910]: Next returned nil segment Nov 5 07:09:15.183553485 (139h28m46.879402239s) [upload=122910]: Waiting for error group managing segments... Nov 5 07:09:15.515060579 (139h28m47.210909332s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.562983674 (139h28m47.258832417s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.584642687 (139h28m47.28049144s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.591557684 (139h28m47.287406427s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.601517589 (139h28m47.297366333s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.688637153 (139h28m47.384485906s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:42.448516803 (139h29m14.144365566s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.448842534 (139h29m14.144691287s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: rpc: dial tcp 145.79.227.13:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449324783 (139h29m14.145173536s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449692342 (139h29m14.145541085s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450039493 (139h29m14.145888236s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450238873 (139h29m14.146087626s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450426109 (139h29m14.146274862s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450710671 (139h29m14.146559434s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: rpc: dial tcp 145.79.227.20:4437: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450894401 (139h29m14.146743154s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: rpc: dial tcp 145.79.227.25:4617: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.451068282 (139h29m14.146917036s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.453469054 (139h29m14.149317797s) [seg_pos={0 0}, upload=122910]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:43.162545170 (139h29m14.858393923s) [upload=122910]: Done uploading segments... Nov 5 07:09:43.162601467 (139h29m14.85845022s) [upload=122910]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 5.94192226s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.645495489s trace: https://jaeger.dev.storj.io/trace/75afb5f91d3935bc FAIL (2025-11-05 07:15:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== v 5 07:09:30.093538136 (139h29m1.789386889s) [upload=122911]: Uploading segments... Nov 5 07:09:30.093576248 (139h29m1.789424992s) [upload=122911]: Got next segment. Inline: false Nov 5 07:09:30.093604072 (139h29m1.789452825s) [upload=122911]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 07:09:30.093608600 (139h29m1.789457343s) [upload=122911]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:30.339289624 (139h29m2.035138367s) [seg_pos={0 0}, upload=122911]: Begin upload segment... Nov 5 07:09:30.339349428 (139h29m2.035198181s) [seg_pos={0 0}, upload=122911]: Done begin upload segment. Nov 5 07:09:30.339386408 (139h29m2.035235151s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340339152 (139h29m2.036187905s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340990803 (139h29m2.036839556s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341444226 (139h29m2.03729298s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341919592 (139h29m2.037768375s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342371553 (139h29m2.038220306s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342835706 (139h29m2.038684449s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343494230 (139h29m2.039342973s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343970537 (139h29m2.03981929s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.345438873 (139h29m2.041287626s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.347000497 (139h29m2.04284925s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.348576918 (139h29m2.044425661s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.350325538 (139h29m2.046174281s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.351367712 (139h29m2.047216465s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.354636787 (139h29m2.050485571s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.356330162 (139h29m2.052178915s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357738744 (139h29m2.053587487s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357956329 (139h29m2.053805172s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:31.727744519 (139h29m3.423593262s) [upload=122911]: Next returned nil segment Nov 5 07:09:31.727759807 (139h29m3.42360856s) [upload=122911]: Waiting for error group managing segments... Nov 5 07:09:32.079494147 (139h29m3.77534289s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.084663190 (139h29m3.780511933s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.140346892 (139h29m3.836195645s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.141263718 (139h29m3.837112472s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.325083005 (139h29m4.020931758s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.327962548 (139h29m4.023811301s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.375450033 (139h29m4.071298786s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.421783842 (139h29m4.117632595s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.435408857 (139h29m4.13125761s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.450128345 (139h29m4.145977088s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.464432925 (139h29m4.160281678s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.483329487 (139h29m4.17917824s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.594589918 (139h29m4.290438661s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:10:29.406200389 (139h30m1.102049142s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:145.79.227.13:4820): protocol: rpc: dial tcp 145.79.227.13:4820: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406293496 (139h30m1.102142249s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb, address:145.79.227.16:4920): protocol: rpc: dial tcp 145.79.227.16:4920: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406452028 (139h30m1.102300781s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:145.79.227.20:4440): protocol: rpc: dial tcp 145.79.227.20:4440: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406646058 (139h30m1.102494801s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:145.79.227.39:4047): protocol: rpc: dial tcp 145.79.227.39:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406963362 (139h30m1.102812125s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: rpc: dial tcp 145.79.227.2:4131: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.407007867 (139h30m1.10285662s) [seg_pos={0 0}, upload=122911]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 5 07:10:29.648039207 (139h30m1.34388797s) [upload=122911]: Done uploading segments... Nov 5 07:10:29.648083852 (139h30m1.343932595s) [upload=122911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.645552908s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.804333422s trace: https://jaeger.dev.storj.io/trace/75afb5f91d3935bc FAIL (2025-11-05 07:15:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== v 5 07:09:30.093538136 (139h29m1.789386889s) [upload=122911]: Uploading segments... Nov 5 07:09:30.093576248 (139h29m1.789424992s) [upload=122911]: Got next segment. Inline: false Nov 5 07:09:30.093604072 (139h29m1.789452825s) [upload=122911]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 07:09:30.093608600 (139h29m1.789457343s) [upload=122911]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:30.339289624 (139h29m2.035138367s) [seg_pos={0 0}, upload=122911]: Begin upload segment... Nov 5 07:09:30.339349428 (139h29m2.035198181s) [seg_pos={0 0}, upload=122911]: Done begin upload segment. Nov 5 07:09:30.339386408 (139h29m2.035235151s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340339152 (139h29m2.036187905s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340990803 (139h29m2.036839556s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341444226 (139h29m2.03729298s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341919592 (139h29m2.037768375s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342371553 (139h29m2.038220306s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342835706 (139h29m2.038684449s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343494230 (139h29m2.039342973s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343970537 (139h29m2.03981929s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.345438873 (139h29m2.041287626s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.347000497 (139h29m2.04284925s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.348576918 (139h29m2.044425661s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.350325538 (139h29m2.046174281s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.351367712 (139h29m2.047216465s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.354636787 (139h29m2.050485571s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.356330162 (139h29m2.052178915s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357738744 (139h29m2.053587487s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357956329 (139h29m2.053805172s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:31.727744519 (139h29m3.423593262s) [upload=122911]: Next returned nil segment Nov 5 07:09:31.727759807 (139h29m3.42360856s) [upload=122911]: Waiting for error group managing segments... Nov 5 07:09:32.079494147 (139h29m3.77534289s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.084663190 (139h29m3.780511933s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.140346892 (139h29m3.836195645s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.141263718 (139h29m3.837112472s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.325083005 (139h29m4.020931758s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.327962548 (139h29m4.023811301s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.375450033 (139h29m4.071298786s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.421783842 (139h29m4.117632595s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.435408857 (139h29m4.13125761s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.450128345 (139h29m4.145977088s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.464432925 (139h29m4.160281678s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.483329487 (139h29m4.17917824s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.594589918 (139h29m4.290438661s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:10:29.406200389 (139h30m1.102049142s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:145.79.227.13:4820): protocol: rpc: dial tcp 145.79.227.13:4820: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406293496 (139h30m1.102142249s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb, address:145.79.227.16:4920): protocol: rpc: dial tcp 145.79.227.16:4920: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406452028 (139h30m1.102300781s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:145.79.227.20:4440): protocol: rpc: dial tcp 145.79.227.20:4440: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406646058 (139h30m1.102494801s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:145.79.227.39:4047): protocol: rpc: dial tcp 145.79.227.39:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406963362 (139h30m1.102812125s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: rpc: dial tcp 145.79.227.2:4131: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.407007867 (139h30m1.10285662s) [seg_pos={0 0}, upload=122911]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 5 07:10:29.648039207 (139h30m1.34388797s) [upload=122911]: Done uploading segments... Nov 5 07:10:29.648083852 (139h30m1.343932595s) [upload=122911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.804367177s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.418415719s trace: https://jaeger.dev.storj.io/trace/1b48011ff2f3bc78 FAIL (2025-11-05 07:14:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.418447009s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................XX......] duration: 5.821791226s trace: https://jaeger.dev.storj.io/trace/fe05d2f05e7af5f FAIL (2025-11-05 07:14:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 164056 (139h28m44.854012819s) [upload=122910]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:13.414175591 (139h28m45.110024344s) [seg_pos={0 0}, upload=122910]: Begin upload segment... Nov 5 07:09:13.414231787 (139h28m45.110080541s) [seg_pos={0 0}, upload=122910]: Done begin upload segment. Nov 5 07:09:13.414274930 (139h28m45.110123683s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415327825 (139h28m45.111176578s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415861491 (139h28m45.111710244s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416375510 (139h28m45.112224263s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416678197 (139h28m45.11252695s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417009026 (139h28m45.112857779s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417341269 (139h28m45.113190022s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417656319 (139h28m45.113505072s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.418114902 (139h28m45.113963645s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.420954530 (139h28m45.116803273s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.422854929 (139h28m45.118703682s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.424915342 (139h28m45.120764095s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.426819858 (139h28m45.122668611s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.428917523 (139h28m45.124766276s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.430615777 (139h28m45.12646452s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.432400986 (139h28m45.128249739s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:15.183539119 (139h28m46.879387872s) [upload=122910]: Next returned nil segment Nov 5 07:09:15.183553485 (139h28m46.879402239s) [upload=122910]: Waiting for error group managing segments... Nov 5 07:09:15.515060579 (139h28m47.210909332s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.562983674 (139h28m47.258832417s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.584642687 (139h28m47.28049144s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.591557684 (139h28m47.287406427s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.601517589 (139h28m47.297366333s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.688637153 (139h28m47.384485906s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:42.448516803 (139h29m14.144365566s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.448842534 (139h29m14.144691287s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: rpc: dial tcp 145.79.227.13:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449324783 (139h29m14.145173536s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449692342 (139h29m14.145541085s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450039493 (139h29m14.145888236s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450238873 (139h29m14.146087626s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450426109 (139h29m14.146274862s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450710671 (139h29m14.146559434s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: rpc: dial tcp 145.79.227.20:4437: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450894401 (139h29m14.146743154s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: rpc: dial tcp 145.79.227.25:4617: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.451068282 (139h29m14.146917036s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.453469054 (139h29m14.149317797s) [seg_pos={0 0}, upload=122910]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:43.162545170 (139h29m14.858393923s) [upload=122910]: Done uploading segments... Nov 5 07:09:43.162601467 (139h29m14.85845022s) [upload=122910]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 5.82182509s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.630152851s trace: https://jaeger.dev.storj.io/trace/75afb5f91d3935bc FAIL (2025-11-05 07:14:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== v 5 07:09:30.093538136 (139h29m1.789386889s) [upload=122911]: Uploading segments... Nov 5 07:09:30.093576248 (139h29m1.789424992s) [upload=122911]: Got next segment. Inline: false Nov 5 07:09:30.093604072 (139h29m1.789452825s) [upload=122911]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 07:09:30.093608600 (139h29m1.789457343s) [upload=122911]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:30.339289624 (139h29m2.035138367s) [seg_pos={0 0}, upload=122911]: Begin upload segment... Nov 5 07:09:30.339349428 (139h29m2.035198181s) [seg_pos={0 0}, upload=122911]: Done begin upload segment. Nov 5 07:09:30.339386408 (139h29m2.035235151s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340339152 (139h29m2.036187905s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340990803 (139h29m2.036839556s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341444226 (139h29m2.03729298s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341919592 (139h29m2.037768375s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342371553 (139h29m2.038220306s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342835706 (139h29m2.038684449s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343494230 (139h29m2.039342973s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343970537 (139h29m2.03981929s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.345438873 (139h29m2.041287626s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.347000497 (139h29m2.04284925s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.348576918 (139h29m2.044425661s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.350325538 (139h29m2.046174281s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.351367712 (139h29m2.047216465s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.354636787 (139h29m2.050485571s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.356330162 (139h29m2.052178915s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357738744 (139h29m2.053587487s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357956329 (139h29m2.053805172s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:31.727744519 (139h29m3.423593262s) [upload=122911]: Next returned nil segment Nov 5 07:09:31.727759807 (139h29m3.42360856s) [upload=122911]: Waiting for error group managing segments... Nov 5 07:09:32.079494147 (139h29m3.77534289s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.084663190 (139h29m3.780511933s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.140346892 (139h29m3.836195645s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.141263718 (139h29m3.837112472s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.325083005 (139h29m4.020931758s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.327962548 (139h29m4.023811301s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.375450033 (139h29m4.071298786s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.421783842 (139h29m4.117632595s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.435408857 (139h29m4.13125761s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.450128345 (139h29m4.145977088s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.464432925 (139h29m4.160281678s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.483329487 (139h29m4.17917824s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.594589918 (139h29m4.290438661s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:10:29.406200389 (139h30m1.102049142s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:145.79.227.13:4820): protocol: rpc: dial tcp 145.79.227.13:4820: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406293496 (139h30m1.102142249s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb, address:145.79.227.16:4920): protocol: rpc: dial tcp 145.79.227.16:4920: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406452028 (139h30m1.102300781s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:145.79.227.20:4440): protocol: rpc: dial tcp 145.79.227.20:4440: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406646058 (139h30m1.102494801s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:145.79.227.39:4047): protocol: rpc: dial tcp 145.79.227.39:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406963362 (139h30m1.102812125s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: rpc: dial tcp 145.79.227.2:4131: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.407007867 (139h30m1.10285662s) [seg_pos={0 0}, upload=122911]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 5 07:10:29.648039207 (139h30m1.34388797s) [upload=122911]: Done uploading segments... Nov 5 07:10:29.648083852 (139h30m1.343932595s) [upload=122911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.630198768s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.152269281s trace: https://jaeger.dev.storj.io/trace/1b48011ff2f3bc78 FAIL (2025-11-05 07:14:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.152369262s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................XX.....] duration: 5.818631582s trace: https://jaeger.dev.storj.io/trace/fe05d2f05e7af5f FAIL (2025-11-05 07:13:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 164056 (139h28m44.854012819s) [upload=122910]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:13.414175591 (139h28m45.110024344s) [seg_pos={0 0}, upload=122910]: Begin upload segment... Nov 5 07:09:13.414231787 (139h28m45.110080541s) [seg_pos={0 0}, upload=122910]: Done begin upload segment. Nov 5 07:09:13.414274930 (139h28m45.110123683s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415327825 (139h28m45.111176578s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415861491 (139h28m45.111710244s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416375510 (139h28m45.112224263s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416678197 (139h28m45.11252695s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417009026 (139h28m45.112857779s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417341269 (139h28m45.113190022s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417656319 (139h28m45.113505072s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.418114902 (139h28m45.113963645s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.420954530 (139h28m45.116803273s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.422854929 (139h28m45.118703682s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.424915342 (139h28m45.120764095s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.426819858 (139h28m45.122668611s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.428917523 (139h28m45.124766276s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.430615777 (139h28m45.12646452s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.432400986 (139h28m45.128249739s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:15.183539119 (139h28m46.879387872s) [upload=122910]: Next returned nil segment Nov 5 07:09:15.183553485 (139h28m46.879402239s) [upload=122910]: Waiting for error group managing segments... Nov 5 07:09:15.515060579 (139h28m47.210909332s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.562983674 (139h28m47.258832417s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.584642687 (139h28m47.28049144s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.591557684 (139h28m47.287406427s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.601517589 (139h28m47.297366333s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.688637153 (139h28m47.384485906s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:42.448516803 (139h29m14.144365566s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.448842534 (139h29m14.144691287s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: rpc: dial tcp 145.79.227.13:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449324783 (139h29m14.145173536s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449692342 (139h29m14.145541085s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450039493 (139h29m14.145888236s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450238873 (139h29m14.146087626s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450426109 (139h29m14.146274862s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450710671 (139h29m14.146559434s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: rpc: dial tcp 145.79.227.20:4437: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450894401 (139h29m14.146743154s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: rpc: dial tcp 145.79.227.25:4617: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.451068282 (139h29m14.146917036s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.453469054 (139h29m14.149317797s) [seg_pos={0 0}, upload=122910]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:43.162545170 (139h29m14.858393923s) [upload=122910]: Done uploading segments... Nov 5 07:09:43.162601467 (139h29m14.85845022s) [upload=122910]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 5.818665486s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.094729597s trace: https://jaeger.dev.storj.io/trace/1b48011ff2f3bc78 FAIL (2025-11-05 07:13:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.094762189s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.056154794s trace: https://jaeger.dev.storj.io/trace/75afb5f91d3935bc FAIL (2025-11-05 07:13:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== v 5 07:09:30.093538136 (139h29m1.789386889s) [upload=122911]: Uploading segments... Nov 5 07:09:30.093576248 (139h29m1.789424992s) [upload=122911]: Got next segment. Inline: false Nov 5 07:09:30.093604072 (139h29m1.789452825s) [upload=122911]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 07:09:30.093608600 (139h29m1.789457343s) [upload=122911]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:30.339289624 (139h29m2.035138367s) [seg_pos={0 0}, upload=122911]: Begin upload segment... Nov 5 07:09:30.339349428 (139h29m2.035198181s) [seg_pos={0 0}, upload=122911]: Done begin upload segment. Nov 5 07:09:30.339386408 (139h29m2.035235151s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340339152 (139h29m2.036187905s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340990803 (139h29m2.036839556s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341444226 (139h29m2.03729298s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341919592 (139h29m2.037768375s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342371553 (139h29m2.038220306s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342835706 (139h29m2.038684449s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343494230 (139h29m2.039342973s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343970537 (139h29m2.03981929s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.345438873 (139h29m2.041287626s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.347000497 (139h29m2.04284925s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.348576918 (139h29m2.044425661s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.350325538 (139h29m2.046174281s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.351367712 (139h29m2.047216465s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.354636787 (139h29m2.050485571s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.356330162 (139h29m2.052178915s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357738744 (139h29m2.053587487s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357956329 (139h29m2.053805172s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:31.727744519 (139h29m3.423593262s) [upload=122911]: Next returned nil segment Nov 5 07:09:31.727759807 (139h29m3.42360856s) [upload=122911]: Waiting for error group managing segments... Nov 5 07:09:32.079494147 (139h29m3.77534289s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.084663190 (139h29m3.780511933s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.140346892 (139h29m3.836195645s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.141263718 (139h29m3.837112472s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.325083005 (139h29m4.020931758s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.327962548 (139h29m4.023811301s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.375450033 (139h29m4.071298786s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.421783842 (139h29m4.117632595s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.435408857 (139h29m4.13125761s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.450128345 (139h29m4.145977088s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.464432925 (139h29m4.160281678s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.483329487 (139h29m4.17917824s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.594589918 (139h29m4.290438661s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:10:29.406200389 (139h30m1.102049142s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:145.79.227.13:4820): protocol: rpc: dial tcp 145.79.227.13:4820: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406293496 (139h30m1.102142249s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb, address:145.79.227.16:4920): protocol: rpc: dial tcp 145.79.227.16:4920: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406452028 (139h30m1.102300781s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:145.79.227.20:4440): protocol: rpc: dial tcp 145.79.227.20:4440: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406646058 (139h30m1.102494801s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:145.79.227.39:4047): protocol: rpc: dial tcp 145.79.227.39:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406963362 (139h30m1.102812125s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: rpc: dial tcp 145.79.227.2:4131: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.407007867 (139h30m1.10285662s) [seg_pos={0 0}, upload=122911]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 5 07:10:29.648039207 (139h30m1.34388797s) [upload=122911]: Done uploading segments... Nov 5 07:10:29.648083852 (139h30m1.343932595s) [upload=122911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.056183979s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.218099688s trace: https://jaeger.dev.storj.io/trace/1b48011ff2f3bc78 FAIL (2025-11-05 07:13:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.218145885s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.185568145s trace: https://jaeger.dev.storj.io/trace/1b48011ff2f3bc78 FAIL (2025-11-05 07:12:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.185599406s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................XX....] duration: 5.846708994s trace: https://jaeger.dev.storj.io/trace/fe05d2f05e7af5f FAIL (2025-11-05 07:12:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 164056 (139h28m44.854012819s) [upload=122910]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:13.414175591 (139h28m45.110024344s) [seg_pos={0 0}, upload=122910]: Begin upload segment... Nov 5 07:09:13.414231787 (139h28m45.110080541s) [seg_pos={0 0}, upload=122910]: Done begin upload segment. Nov 5 07:09:13.414274930 (139h28m45.110123683s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415327825 (139h28m45.111176578s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415861491 (139h28m45.111710244s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416375510 (139h28m45.112224263s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416678197 (139h28m45.11252695s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417009026 (139h28m45.112857779s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417341269 (139h28m45.113190022s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417656319 (139h28m45.113505072s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.418114902 (139h28m45.113963645s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.420954530 (139h28m45.116803273s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.422854929 (139h28m45.118703682s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.424915342 (139h28m45.120764095s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.426819858 (139h28m45.122668611s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.428917523 (139h28m45.124766276s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.430615777 (139h28m45.12646452s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.432400986 (139h28m45.128249739s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:15.183539119 (139h28m46.879387872s) [upload=122910]: Next returned nil segment Nov 5 07:09:15.183553485 (139h28m46.879402239s) [upload=122910]: Waiting for error group managing segments... Nov 5 07:09:15.515060579 (139h28m47.210909332s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.562983674 (139h28m47.258832417s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.584642687 (139h28m47.28049144s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.591557684 (139h28m47.287406427s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.601517589 (139h28m47.297366333s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.688637153 (139h28m47.384485906s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:42.448516803 (139h29m14.144365566s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.448842534 (139h29m14.144691287s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: rpc: dial tcp 145.79.227.13:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449324783 (139h29m14.145173536s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449692342 (139h29m14.145541085s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450039493 (139h29m14.145888236s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450238873 (139h29m14.146087626s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450426109 (139h29m14.146274862s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450710671 (139h29m14.146559434s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: rpc: dial tcp 145.79.227.20:4437: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450894401 (139h29m14.146743154s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: rpc: dial tcp 145.79.227.25:4617: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.451068282 (139h29m14.146917036s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.453469054 (139h29m14.149317797s) [seg_pos={0 0}, upload=122910]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:43.162545170 (139h29m14.858393923s) [upload=122910]: Done uploading segments... Nov 5 07:09:43.162601467 (139h29m14.85845022s) [upload=122910]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 5.846731957s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.762238306s trace: https://jaeger.dev.storj.io/trace/75afb5f91d3935bc FAIL (2025-11-05 07:12:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== v 5 07:09:30.093538136 (139h29m1.789386889s) [upload=122911]: Uploading segments... Nov 5 07:09:30.093576248 (139h29m1.789424992s) [upload=122911]: Got next segment. Inline: false Nov 5 07:09:30.093604072 (139h29m1.789452825s) [upload=122911]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 07:09:30.093608600 (139h29m1.789457343s) [upload=122911]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:30.339289624 (139h29m2.035138367s) [seg_pos={0 0}, upload=122911]: Begin upload segment... Nov 5 07:09:30.339349428 (139h29m2.035198181s) [seg_pos={0 0}, upload=122911]: Done begin upload segment. Nov 5 07:09:30.339386408 (139h29m2.035235151s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340339152 (139h29m2.036187905s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340990803 (139h29m2.036839556s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341444226 (139h29m2.03729298s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341919592 (139h29m2.037768375s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342371553 (139h29m2.038220306s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342835706 (139h29m2.038684449s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343494230 (139h29m2.039342973s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343970537 (139h29m2.03981929s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.345438873 (139h29m2.041287626s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.347000497 (139h29m2.04284925s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.348576918 (139h29m2.044425661s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.350325538 (139h29m2.046174281s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.351367712 (139h29m2.047216465s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.354636787 (139h29m2.050485571s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.356330162 (139h29m2.052178915s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357738744 (139h29m2.053587487s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357956329 (139h29m2.053805172s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:31.727744519 (139h29m3.423593262s) [upload=122911]: Next returned nil segment Nov 5 07:09:31.727759807 (139h29m3.42360856s) [upload=122911]: Waiting for error group managing segments... Nov 5 07:09:32.079494147 (139h29m3.77534289s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.084663190 (139h29m3.780511933s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.140346892 (139h29m3.836195645s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.141263718 (139h29m3.837112472s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.325083005 (139h29m4.020931758s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.327962548 (139h29m4.023811301s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.375450033 (139h29m4.071298786s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.421783842 (139h29m4.117632595s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.435408857 (139h29m4.13125761s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.450128345 (139h29m4.145977088s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.464432925 (139h29m4.160281678s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.483329487 (139h29m4.17917824s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.594589918 (139h29m4.290438661s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:10:29.406200389 (139h30m1.102049142s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:145.79.227.13:4820): protocol: rpc: dial tcp 145.79.227.13:4820: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406293496 (139h30m1.102142249s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb, address:145.79.227.16:4920): protocol: rpc: dial tcp 145.79.227.16:4920: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406452028 (139h30m1.102300781s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:145.79.227.20:4440): protocol: rpc: dial tcp 145.79.227.20:4440: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406646058 (139h30m1.102494801s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:145.79.227.39:4047): protocol: rpc: dial tcp 145.79.227.39:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406963362 (139h30m1.102812125s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: rpc: dial tcp 145.79.227.2:4131: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.407007867 (139h30m1.10285662s) [seg_pos={0 0}, upload=122911]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 5 07:10:29.648039207 (139h30m1.34388797s) [upload=122911]: Done uploading segments... Nov 5 07:10:29.648083852 (139h30m1.343932595s) [upload=122911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.762268153s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................XX...] duration: 6.086274785s trace: https://jaeger.dev.storj.io/trace/fe05d2f05e7af5f FAIL (2025-11-05 07:11:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 164056 (139h28m44.854012819s) [upload=122910]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:13.414175591 (139h28m45.110024344s) [seg_pos={0 0}, upload=122910]: Begin upload segment... Nov 5 07:09:13.414231787 (139h28m45.110080541s) [seg_pos={0 0}, upload=122910]: Done begin upload segment. Nov 5 07:09:13.414274930 (139h28m45.110123683s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415327825 (139h28m45.111176578s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415861491 (139h28m45.111710244s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416375510 (139h28m45.112224263s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416678197 (139h28m45.11252695s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417009026 (139h28m45.112857779s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417341269 (139h28m45.113190022s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417656319 (139h28m45.113505072s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.418114902 (139h28m45.113963645s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.420954530 (139h28m45.116803273s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.422854929 (139h28m45.118703682s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.424915342 (139h28m45.120764095s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.426819858 (139h28m45.122668611s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.428917523 (139h28m45.124766276s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.430615777 (139h28m45.12646452s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.432400986 (139h28m45.128249739s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:15.183539119 (139h28m46.879387872s) [upload=122910]: Next returned nil segment Nov 5 07:09:15.183553485 (139h28m46.879402239s) [upload=122910]: Waiting for error group managing segments... Nov 5 07:09:15.515060579 (139h28m47.210909332s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.562983674 (139h28m47.258832417s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.584642687 (139h28m47.28049144s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.591557684 (139h28m47.287406427s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.601517589 (139h28m47.297366333s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.688637153 (139h28m47.384485906s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:42.448516803 (139h29m14.144365566s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.448842534 (139h29m14.144691287s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: rpc: dial tcp 145.79.227.13:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449324783 (139h29m14.145173536s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449692342 (139h29m14.145541085s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450039493 (139h29m14.145888236s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450238873 (139h29m14.146087626s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450426109 (139h29m14.146274862s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450710671 (139h29m14.146559434s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: rpc: dial tcp 145.79.227.20:4437: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450894401 (139h29m14.146743154s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: rpc: dial tcp 145.79.227.25:4617: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.451068282 (139h29m14.146917036s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.453469054 (139h29m14.149317797s) [seg_pos={0 0}, upload=122910]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:43.162545170 (139h29m14.858393923s) [upload=122910]: Done uploading segments... Nov 5 07:09:43.162601467 (139h29m14.85845022s) [upload=122910]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 6.086299572s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.073907425s trace: https://jaeger.dev.storj.io/trace/1b48011ff2f3bc78 FAIL (2025-11-05 07:11:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.073934685s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.644110548s trace: https://jaeger.dev.storj.io/trace/75afb5f91d3935bc FAIL (2025-11-05 07:11:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== v 5 07:09:30.093538136 (139h29m1.789386889s) [upload=122911]: Uploading segments... Nov 5 07:09:30.093576248 (139h29m1.789424992s) [upload=122911]: Got next segment. Inline: false Nov 5 07:09:30.093604072 (139h29m1.789452825s) [upload=122911]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 07:09:30.093608600 (139h29m1.789457343s) [upload=122911]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:30.339289624 (139h29m2.035138367s) [seg_pos={0 0}, upload=122911]: Begin upload segment... Nov 5 07:09:30.339349428 (139h29m2.035198181s) [seg_pos={0 0}, upload=122911]: Done begin upload segment. Nov 5 07:09:30.339386408 (139h29m2.035235151s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340339152 (139h29m2.036187905s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340990803 (139h29m2.036839556s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341444226 (139h29m2.03729298s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341919592 (139h29m2.037768375s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342371553 (139h29m2.038220306s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342835706 (139h29m2.038684449s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343494230 (139h29m2.039342973s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343970537 (139h29m2.03981929s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.345438873 (139h29m2.041287626s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.347000497 (139h29m2.04284925s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.348576918 (139h29m2.044425661s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.350325538 (139h29m2.046174281s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.351367712 (139h29m2.047216465s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.354636787 (139h29m2.050485571s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.356330162 (139h29m2.052178915s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357738744 (139h29m2.053587487s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357956329 (139h29m2.053805172s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:31.727744519 (139h29m3.423593262s) [upload=122911]: Next returned nil segment Nov 5 07:09:31.727759807 (139h29m3.42360856s) [upload=122911]: Waiting for error group managing segments... Nov 5 07:09:32.079494147 (139h29m3.77534289s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.084663190 (139h29m3.780511933s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.140346892 (139h29m3.836195645s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.141263718 (139h29m3.837112472s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.325083005 (139h29m4.020931758s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.327962548 (139h29m4.023811301s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.375450033 (139h29m4.071298786s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.421783842 (139h29m4.117632595s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.435408857 (139h29m4.13125761s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.450128345 (139h29m4.145977088s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.464432925 (139h29m4.160281678s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.483329487 (139h29m4.17917824s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.594589918 (139h29m4.290438661s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:10:29.406200389 (139h30m1.102049142s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:145.79.227.13:4820): protocol: rpc: dial tcp 145.79.227.13:4820: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406293496 (139h30m1.102142249s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb, address:145.79.227.16:4920): protocol: rpc: dial tcp 145.79.227.16:4920: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406452028 (139h30m1.102300781s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:145.79.227.20:4440): protocol: rpc: dial tcp 145.79.227.20:4440: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406646058 (139h30m1.102494801s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:145.79.227.39:4047): protocol: rpc: dial tcp 145.79.227.39:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406963362 (139h30m1.102812125s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: rpc: dial tcp 145.79.227.2:4131: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.407007867 (139h30m1.10285662s) [seg_pos={0 0}, upload=122911]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 5 07:10:29.648039207 (139h30m1.34388797s) [upload=122911]: Done uploading segments... Nov 5 07:10:29.648083852 (139h30m1.343932595s) [upload=122911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.644141648s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.877646309s trace: https://jaeger.dev.storj.io/trace/75afb5f91d3935bc FAIL (2025-11-05 07:10:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== v 5 07:09:30.093538136 (139h29m1.789386889s) [upload=122911]: Uploading segments... Nov 5 07:09:30.093576248 (139h29m1.789424992s) [upload=122911]: Got next segment. Inline: false Nov 5 07:09:30.093604072 (139h29m1.789452825s) [upload=122911]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 07:09:30.093608600 (139h29m1.789457343s) [upload=122911]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:30.339289624 (139h29m2.035138367s) [seg_pos={0 0}, upload=122911]: Begin upload segment... Nov 5 07:09:30.339349428 (139h29m2.035198181s) [seg_pos={0 0}, upload=122911]: Done begin upload segment. Nov 5 07:09:30.339386408 (139h29m2.035235151s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340339152 (139h29m2.036187905s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340990803 (139h29m2.036839556s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341444226 (139h29m2.03729298s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341919592 (139h29m2.037768375s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342371553 (139h29m2.038220306s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342835706 (139h29m2.038684449s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343494230 (139h29m2.039342973s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343970537 (139h29m2.03981929s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.345438873 (139h29m2.041287626s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.347000497 (139h29m2.04284925s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.348576918 (139h29m2.044425661s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.350325538 (139h29m2.046174281s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.351367712 (139h29m2.047216465s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.354636787 (139h29m2.050485571s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.356330162 (139h29m2.052178915s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357738744 (139h29m2.053587487s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357956329 (139h29m2.053805172s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:31.727744519 (139h29m3.423593262s) [upload=122911]: Next returned nil segment Nov 5 07:09:31.727759807 (139h29m3.42360856s) [upload=122911]: Waiting for error group managing segments... Nov 5 07:09:32.079494147 (139h29m3.77534289s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.084663190 (139h29m3.780511933s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.140346892 (139h29m3.836195645s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.141263718 (139h29m3.837112472s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.325083005 (139h29m4.020931758s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.327962548 (139h29m4.023811301s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.375450033 (139h29m4.071298786s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.421783842 (139h29m4.117632595s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.435408857 (139h29m4.13125761s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.450128345 (139h29m4.145977088s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.464432925 (139h29m4.160281678s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.483329487 (139h29m4.17917824s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.594589918 (139h29m4.290438661s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:10:29.406200389 (139h30m1.102049142s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:145.79.227.13:4820): protocol: rpc: dial tcp 145.79.227.13:4820: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406293496 (139h30m1.102142249s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb, address:145.79.227.16:4920): protocol: rpc: dial tcp 145.79.227.16:4920: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406452028 (139h30m1.102300781s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:145.79.227.20:4440): protocol: rpc: dial tcp 145.79.227.20:4440: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406646058 (139h30m1.102494801s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:145.79.227.39:4047): protocol: rpc: dial tcp 145.79.227.39:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406963362 (139h30m1.102812125s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: rpc: dial tcp 145.79.227.2:4131: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.407007867 (139h30m1.10285662s) [seg_pos={0 0}, upload=122911]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 5 07:10:29.648039207 (139h30m1.34388797s) [upload=122911]: Done uploading segments... Nov 5 07:10:29.648083852 (139h30m1.343932595s) [upload=122911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.877680665s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)