Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......XX...................................................] duration: 5.723695719s trace: https://jaeger.dev.storj.io/trace/fe05d2f05e7af5f FAIL (2025-11-05 07:51:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 164056 (139h28m44.854012819s) [upload=122910]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:13.414175591 (139h28m45.110024344s) [seg_pos={0 0}, upload=122910]: Begin upload segment... Nov 5 07:09:13.414231787 (139h28m45.110080541s) [seg_pos={0 0}, upload=122910]: Done begin upload segment. Nov 5 07:09:13.414274930 (139h28m45.110123683s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415327825 (139h28m45.111176578s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415861491 (139h28m45.111710244s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416375510 (139h28m45.112224263s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416678197 (139h28m45.11252695s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417009026 (139h28m45.112857779s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417341269 (139h28m45.113190022s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417656319 (139h28m45.113505072s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.418114902 (139h28m45.113963645s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.420954530 (139h28m45.116803273s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.422854929 (139h28m45.118703682s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.424915342 (139h28m45.120764095s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.426819858 (139h28m45.122668611s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.428917523 (139h28m45.124766276s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.430615777 (139h28m45.12646452s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.432400986 (139h28m45.128249739s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:15.183539119 (139h28m46.879387872s) [upload=122910]: Next returned nil segment Nov 5 07:09:15.183553485 (139h28m46.879402239s) [upload=122910]: Waiting for error group managing segments... Nov 5 07:09:15.515060579 (139h28m47.210909332s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.562983674 (139h28m47.258832417s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.584642687 (139h28m47.28049144s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.591557684 (139h28m47.287406427s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.601517589 (139h28m47.297366333s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.688637153 (139h28m47.384485906s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:42.448516803 (139h29m14.144365566s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.448842534 (139h29m14.144691287s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: rpc: dial tcp 145.79.227.13:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449324783 (139h29m14.145173536s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449692342 (139h29m14.145541085s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450039493 (139h29m14.145888236s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450238873 (139h29m14.146087626s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450426109 (139h29m14.146274862s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450710671 (139h29m14.146559434s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: rpc: dial tcp 145.79.227.20:4437: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450894401 (139h29m14.146743154s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: rpc: dial tcp 145.79.227.25:4617: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.451068282 (139h29m14.146917036s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.453469054 (139h29m14.149317797s) [seg_pos={0 0}, upload=122910]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:43.162545170 (139h29m14.858393923s) [upload=122910]: Done uploading segments... Nov 5 07:09:43.162601467 (139h29m14.85845022s) [upload=122910]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 5.723736526s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.818959963s trace: https://jaeger.dev.storj.io/trace/75afb5f91d3935bc FAIL (2025-11-05 07:51:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== v 5 07:09:30.093538136 (139h29m1.789386889s) [upload=122911]: Uploading segments... Nov 5 07:09:30.093576248 (139h29m1.789424992s) [upload=122911]: Got next segment. Inline: false Nov 5 07:09:30.093604072 (139h29m1.789452825s) [upload=122911]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 07:09:30.093608600 (139h29m1.789457343s) [upload=122911]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:30.339289624 (139h29m2.035138367s) [seg_pos={0 0}, upload=122911]: Begin upload segment... Nov 5 07:09:30.339349428 (139h29m2.035198181s) [seg_pos={0 0}, upload=122911]: Done begin upload segment. Nov 5 07:09:30.339386408 (139h29m2.035235151s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340339152 (139h29m2.036187905s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340990803 (139h29m2.036839556s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341444226 (139h29m2.03729298s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341919592 (139h29m2.037768375s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342371553 (139h29m2.038220306s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342835706 (139h29m2.038684449s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343494230 (139h29m2.039342973s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343970537 (139h29m2.03981929s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.345438873 (139h29m2.041287626s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.347000497 (139h29m2.04284925s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.348576918 (139h29m2.044425661s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.350325538 (139h29m2.046174281s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.351367712 (139h29m2.047216465s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.354636787 (139h29m2.050485571s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.356330162 (139h29m2.052178915s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357738744 (139h29m2.053587487s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357956329 (139h29m2.053805172s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:31.727744519 (139h29m3.423593262s) [upload=122911]: Next returned nil segment Nov 5 07:09:31.727759807 (139h29m3.42360856s) [upload=122911]: Waiting for error group managing segments... Nov 5 07:09:32.079494147 (139h29m3.77534289s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.084663190 (139h29m3.780511933s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.140346892 (139h29m3.836195645s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.141263718 (139h29m3.837112472s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.325083005 (139h29m4.020931758s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.327962548 (139h29m4.023811301s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.375450033 (139h29m4.071298786s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.421783842 (139h29m4.117632595s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.435408857 (139h29m4.13125761s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.450128345 (139h29m4.145977088s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.464432925 (139h29m4.160281678s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.483329487 (139h29m4.17917824s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.594589918 (139h29m4.290438661s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:10:29.406200389 (139h30m1.102049142s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:145.79.227.13:4820): protocol: rpc: dial tcp 145.79.227.13:4820: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406293496 (139h30m1.102142249s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb, address:145.79.227.16:4920): protocol: rpc: dial tcp 145.79.227.16:4920: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406452028 (139h30m1.102300781s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:145.79.227.20:4440): protocol: rpc: dial tcp 145.79.227.20:4440: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406646058 (139h30m1.102494801s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:145.79.227.39:4047): protocol: rpc: dial tcp 145.79.227.39:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406963362 (139h30m1.102812125s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: rpc: dial tcp 145.79.227.2:4131: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.407007867 (139h30m1.10285662s) [seg_pos={0 0}, upload=122911]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 5 07:10:29.648039207 (139h30m1.34388797s) [upload=122911]: Done uploading segments... Nov 5 07:10:29.648083852 (139h30m1.343932595s) [upload=122911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.818995401s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........XX..................................................] duration: 5.642979168s trace: https://jaeger.dev.storj.io/trace/fe05d2f05e7af5f FAIL (2025-11-05 07:50:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 164056 (139h28m44.854012819s) [upload=122910]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:13.414175591 (139h28m45.110024344s) [seg_pos={0 0}, upload=122910]: Begin upload segment... Nov 5 07:09:13.414231787 (139h28m45.110080541s) [seg_pos={0 0}, upload=122910]: Done begin upload segment. Nov 5 07:09:13.414274930 (139h28m45.110123683s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415327825 (139h28m45.111176578s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415861491 (139h28m45.111710244s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416375510 (139h28m45.112224263s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416678197 (139h28m45.11252695s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417009026 (139h28m45.112857779s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417341269 (139h28m45.113190022s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417656319 (139h28m45.113505072s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.418114902 (139h28m45.113963645s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.420954530 (139h28m45.116803273s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.422854929 (139h28m45.118703682s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.424915342 (139h28m45.120764095s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.426819858 (139h28m45.122668611s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.428917523 (139h28m45.124766276s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.430615777 (139h28m45.12646452s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.432400986 (139h28m45.128249739s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:15.183539119 (139h28m46.879387872s) [upload=122910]: Next returned nil segment Nov 5 07:09:15.183553485 (139h28m46.879402239s) [upload=122910]: Waiting for error group managing segments... Nov 5 07:09:15.515060579 (139h28m47.210909332s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.562983674 (139h28m47.258832417s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.584642687 (139h28m47.28049144s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.591557684 (139h28m47.287406427s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.601517589 (139h28m47.297366333s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.688637153 (139h28m47.384485906s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:42.448516803 (139h29m14.144365566s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.448842534 (139h29m14.144691287s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: rpc: dial tcp 145.79.227.13:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449324783 (139h29m14.145173536s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449692342 (139h29m14.145541085s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450039493 (139h29m14.145888236s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450238873 (139h29m14.146087626s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450426109 (139h29m14.146274862s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450710671 (139h29m14.146559434s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: rpc: dial tcp 145.79.227.20:4437: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450894401 (139h29m14.146743154s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: rpc: dial tcp 145.79.227.25:4617: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.451068282 (139h29m14.146917036s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.453469054 (139h29m14.149317797s) [seg_pos={0 0}, upload=122910]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:43.162545170 (139h29m14.858393923s) [upload=122910]: Done uploading segments... Nov 5 07:09:43.162601467 (139h29m14.85845022s) [upload=122910]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 5.643011922s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.878728913s trace: https://jaeger.dev.storj.io/trace/75afb5f91d3935bc FAIL (2025-11-05 07:50:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== v 5 07:09:30.093538136 (139h29m1.789386889s) [upload=122911]: Uploading segments... Nov 5 07:09:30.093576248 (139h29m1.789424992s) [upload=122911]: Got next segment. Inline: false Nov 5 07:09:30.093604072 (139h29m1.789452825s) [upload=122911]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 07:09:30.093608600 (139h29m1.789457343s) [upload=122911]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:30.339289624 (139h29m2.035138367s) [seg_pos={0 0}, upload=122911]: Begin upload segment... Nov 5 07:09:30.339349428 (139h29m2.035198181s) [seg_pos={0 0}, upload=122911]: Done begin upload segment. Nov 5 07:09:30.339386408 (139h29m2.035235151s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340339152 (139h29m2.036187905s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340990803 (139h29m2.036839556s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341444226 (139h29m2.03729298s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341919592 (139h29m2.037768375s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342371553 (139h29m2.038220306s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342835706 (139h29m2.038684449s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343494230 (139h29m2.039342973s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343970537 (139h29m2.03981929s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.345438873 (139h29m2.041287626s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.347000497 (139h29m2.04284925s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.348576918 (139h29m2.044425661s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.350325538 (139h29m2.046174281s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.351367712 (139h29m2.047216465s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.354636787 (139h29m2.050485571s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.356330162 (139h29m2.052178915s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357738744 (139h29m2.053587487s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357956329 (139h29m2.053805172s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:31.727744519 (139h29m3.423593262s) [upload=122911]: Next returned nil segment Nov 5 07:09:31.727759807 (139h29m3.42360856s) [upload=122911]: Waiting for error group managing segments... Nov 5 07:09:32.079494147 (139h29m3.77534289s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.084663190 (139h29m3.780511933s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.140346892 (139h29m3.836195645s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.141263718 (139h29m3.837112472s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.325083005 (139h29m4.020931758s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.327962548 (139h29m4.023811301s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.375450033 (139h29m4.071298786s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.421783842 (139h29m4.117632595s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.435408857 (139h29m4.13125761s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.450128345 (139h29m4.145977088s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.464432925 (139h29m4.160281678s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.483329487 (139h29m4.17917824s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.594589918 (139h29m4.290438661s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:10:29.406200389 (139h30m1.102049142s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:145.79.227.13:4820): protocol: rpc: dial tcp 145.79.227.13:4820: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406293496 (139h30m1.102142249s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb, address:145.79.227.16:4920): protocol: rpc: dial tcp 145.79.227.16:4920: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406452028 (139h30m1.102300781s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:145.79.227.20:4440): protocol: rpc: dial tcp 145.79.227.20:4440: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406646058 (139h30m1.102494801s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:145.79.227.39:4047): protocol: rpc: dial tcp 145.79.227.39:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406963362 (139h30m1.102812125s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: rpc: dial tcp 145.79.227.2:4131: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.407007867 (139h30m1.10285662s) [seg_pos={0 0}, upload=122911]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 5 07:10:29.648039207 (139h30m1.34388797s) [upload=122911]: Done uploading segments... Nov 5 07:10:29.648083852 (139h30m1.343932595s) [upload=122911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.878759441s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........XX.................................................] duration: 5.757985694s trace: https://jaeger.dev.storj.io/trace/fe05d2f05e7af5f FAIL (2025-11-05 07:49:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 164056 (139h28m44.854012819s) [upload=122910]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:13.414175591 (139h28m45.110024344s) [seg_pos={0 0}, upload=122910]: Begin upload segment... Nov 5 07:09:13.414231787 (139h28m45.110080541s) [seg_pos={0 0}, upload=122910]: Done begin upload segment. Nov 5 07:09:13.414274930 (139h28m45.110123683s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415327825 (139h28m45.111176578s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415861491 (139h28m45.111710244s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416375510 (139h28m45.112224263s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416678197 (139h28m45.11252695s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417009026 (139h28m45.112857779s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417341269 (139h28m45.113190022s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417656319 (139h28m45.113505072s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.418114902 (139h28m45.113963645s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.420954530 (139h28m45.116803273s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.422854929 (139h28m45.118703682s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.424915342 (139h28m45.120764095s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.426819858 (139h28m45.122668611s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.428917523 (139h28m45.124766276s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.430615777 (139h28m45.12646452s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.432400986 (139h28m45.128249739s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:15.183539119 (139h28m46.879387872s) [upload=122910]: Next returned nil segment Nov 5 07:09:15.183553485 (139h28m46.879402239s) [upload=122910]: Waiting for error group managing segments... Nov 5 07:09:15.515060579 (139h28m47.210909332s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.562983674 (139h28m47.258832417s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.584642687 (139h28m47.28049144s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.591557684 (139h28m47.287406427s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.601517589 (139h28m47.297366333s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.688637153 (139h28m47.384485906s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:42.448516803 (139h29m14.144365566s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.448842534 (139h29m14.144691287s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: rpc: dial tcp 145.79.227.13:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449324783 (139h29m14.145173536s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449692342 (139h29m14.145541085s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450039493 (139h29m14.145888236s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450238873 (139h29m14.146087626s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450426109 (139h29m14.146274862s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450710671 (139h29m14.146559434s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: rpc: dial tcp 145.79.227.20:4437: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450894401 (139h29m14.146743154s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: rpc: dial tcp 145.79.227.25:4617: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.451068282 (139h29m14.146917036s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.453469054 (139h29m14.149317797s) [seg_pos={0 0}, upload=122910]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:43.162545170 (139h29m14.858393923s) [upload=122910]: Done uploading segments... Nov 5 07:09:43.162601467 (139h29m14.85845022s) [upload=122910]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 5.758013406s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.631456007s trace: https://jaeger.dev.storj.io/trace/75afb5f91d3935bc FAIL (2025-11-05 07:49:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== v 5 07:09:30.093538136 (139h29m1.789386889s) [upload=122911]: Uploading segments... Nov 5 07:09:30.093576248 (139h29m1.789424992s) [upload=122911]: Got next segment. Inline: false Nov 5 07:09:30.093604072 (139h29m1.789452825s) [upload=122911]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 07:09:30.093608600 (139h29m1.789457343s) [upload=122911]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:30.339289624 (139h29m2.035138367s) [seg_pos={0 0}, upload=122911]: Begin upload segment... Nov 5 07:09:30.339349428 (139h29m2.035198181s) [seg_pos={0 0}, upload=122911]: Done begin upload segment. Nov 5 07:09:30.339386408 (139h29m2.035235151s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340339152 (139h29m2.036187905s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340990803 (139h29m2.036839556s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341444226 (139h29m2.03729298s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341919592 (139h29m2.037768375s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342371553 (139h29m2.038220306s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342835706 (139h29m2.038684449s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343494230 (139h29m2.039342973s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343970537 (139h29m2.03981929s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.345438873 (139h29m2.041287626s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.347000497 (139h29m2.04284925s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.348576918 (139h29m2.044425661s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.350325538 (139h29m2.046174281s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.351367712 (139h29m2.047216465s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.354636787 (139h29m2.050485571s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.356330162 (139h29m2.052178915s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357738744 (139h29m2.053587487s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357956329 (139h29m2.053805172s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:31.727744519 (139h29m3.423593262s) [upload=122911]: Next returned nil segment Nov 5 07:09:31.727759807 (139h29m3.42360856s) [upload=122911]: Waiting for error group managing segments... Nov 5 07:09:32.079494147 (139h29m3.77534289s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.084663190 (139h29m3.780511933s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.140346892 (139h29m3.836195645s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.141263718 (139h29m3.837112472s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.325083005 (139h29m4.020931758s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.327962548 (139h29m4.023811301s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.375450033 (139h29m4.071298786s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.421783842 (139h29m4.117632595s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.435408857 (139h29m4.13125761s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.450128345 (139h29m4.145977088s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.464432925 (139h29m4.160281678s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.483329487 (139h29m4.17917824s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.594589918 (139h29m4.290438661s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:10:29.406200389 (139h30m1.102049142s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:145.79.227.13:4820): protocol: rpc: dial tcp 145.79.227.13:4820: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406293496 (139h30m1.102142249s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb, address:145.79.227.16:4920): protocol: rpc: dial tcp 145.79.227.16:4920: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406452028 (139h30m1.102300781s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:145.79.227.20:4440): protocol: rpc: dial tcp 145.79.227.20:4440: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406646058 (139h30m1.102494801s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:145.79.227.39:4047): protocol: rpc: dial tcp 145.79.227.39:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406963362 (139h30m1.102812125s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: rpc: dial tcp 145.79.227.2:4131: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.407007867 (139h30m1.10285662s) [seg_pos={0 0}, upload=122911]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 5 07:10:29.648039207 (139h30m1.34388797s) [upload=122911]: Done uploading segments... Nov 5 07:10:29.648083852 (139h30m1.343932595s) [upload=122911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.631477527s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.96691101s trace: https://jaeger.dev.storj.io/trace/75afb5f91d3935bc FAIL (2025-11-05 07:48:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== v 5 07:09:30.093538136 (139h29m1.789386889s) [upload=122911]: Uploading segments... Nov 5 07:09:30.093576248 (139h29m1.789424992s) [upload=122911]: Got next segment. Inline: false Nov 5 07:09:30.093604072 (139h29m1.789452825s) [upload=122911]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 07:09:30.093608600 (139h29m1.789457343s) [upload=122911]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:30.339289624 (139h29m2.035138367s) [seg_pos={0 0}, upload=122911]: Begin upload segment... Nov 5 07:09:30.339349428 (139h29m2.035198181s) [seg_pos={0 0}, upload=122911]: Done begin upload segment. Nov 5 07:09:30.339386408 (139h29m2.035235151s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340339152 (139h29m2.036187905s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340990803 (139h29m2.036839556s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341444226 (139h29m2.03729298s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341919592 (139h29m2.037768375s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342371553 (139h29m2.038220306s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342835706 (139h29m2.038684449s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343494230 (139h29m2.039342973s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343970537 (139h29m2.03981929s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.345438873 (139h29m2.041287626s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.347000497 (139h29m2.04284925s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.348576918 (139h29m2.044425661s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.350325538 (139h29m2.046174281s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.351367712 (139h29m2.047216465s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.354636787 (139h29m2.050485571s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.356330162 (139h29m2.052178915s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357738744 (139h29m2.053587487s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357956329 (139h29m2.053805172s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:31.727744519 (139h29m3.423593262s) [upload=122911]: Next returned nil segment Nov 5 07:09:31.727759807 (139h29m3.42360856s) [upload=122911]: Waiting for error group managing segments... Nov 5 07:09:32.079494147 (139h29m3.77534289s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.084663190 (139h29m3.780511933s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.140346892 (139h29m3.836195645s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.141263718 (139h29m3.837112472s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.325083005 (139h29m4.020931758s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.327962548 (139h29m4.023811301s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.375450033 (139h29m4.071298786s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.421783842 (139h29m4.117632595s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.435408857 (139h29m4.13125761s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.450128345 (139h29m4.145977088s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.464432925 (139h29m4.160281678s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.483329487 (139h29m4.17917824s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.594589918 (139h29m4.290438661s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:10:29.406200389 (139h30m1.102049142s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:145.79.227.13:4820): protocol: rpc: dial tcp 145.79.227.13:4820: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406293496 (139h30m1.102142249s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb, address:145.79.227.16:4920): protocol: rpc: dial tcp 145.79.227.16:4920: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406452028 (139h30m1.102300781s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:145.79.227.20:4440): protocol: rpc: dial tcp 145.79.227.20:4440: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406646058 (139h30m1.102494801s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:145.79.227.39:4047): protocol: rpc: dial tcp 145.79.227.39:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406963362 (139h30m1.102812125s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: rpc: dial tcp 145.79.227.2:4131: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.407007867 (139h30m1.10285662s) [seg_pos={0 0}, upload=122911]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 5 07:10:29.648039207 (139h30m1.34388797s) [upload=122911]: Done uploading segments... Nov 5 07:10:29.648083852 (139h30m1.343932595s) [upload=122911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.966933322s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........XX................................................] duration: 6.016572616s trace: https://jaeger.dev.storj.io/trace/fe05d2f05e7af5f FAIL (2025-11-05 07:48:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 164056 (139h28m44.854012819s) [upload=122910]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:13.414175591 (139h28m45.110024344s) [seg_pos={0 0}, upload=122910]: Begin upload segment... Nov 5 07:09:13.414231787 (139h28m45.110080541s) [seg_pos={0 0}, upload=122910]: Done begin upload segment. Nov 5 07:09:13.414274930 (139h28m45.110123683s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415327825 (139h28m45.111176578s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415861491 (139h28m45.111710244s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416375510 (139h28m45.112224263s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416678197 (139h28m45.11252695s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417009026 (139h28m45.112857779s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417341269 (139h28m45.113190022s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417656319 (139h28m45.113505072s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.418114902 (139h28m45.113963645s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.420954530 (139h28m45.116803273s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.422854929 (139h28m45.118703682s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.424915342 (139h28m45.120764095s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.426819858 (139h28m45.122668611s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.428917523 (139h28m45.124766276s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.430615777 (139h28m45.12646452s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.432400986 (139h28m45.128249739s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:15.183539119 (139h28m46.879387872s) [upload=122910]: Next returned nil segment Nov 5 07:09:15.183553485 (139h28m46.879402239s) [upload=122910]: Waiting for error group managing segments... Nov 5 07:09:15.515060579 (139h28m47.210909332s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.562983674 (139h28m47.258832417s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.584642687 (139h28m47.28049144s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.591557684 (139h28m47.287406427s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.601517589 (139h28m47.297366333s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.688637153 (139h28m47.384485906s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:42.448516803 (139h29m14.144365566s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.448842534 (139h29m14.144691287s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: rpc: dial tcp 145.79.227.13:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449324783 (139h29m14.145173536s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449692342 (139h29m14.145541085s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450039493 (139h29m14.145888236s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450238873 (139h29m14.146087626s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450426109 (139h29m14.146274862s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450710671 (139h29m14.146559434s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: rpc: dial tcp 145.79.227.20:4437: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450894401 (139h29m14.146743154s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: rpc: dial tcp 145.79.227.25:4617: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.451068282 (139h29m14.146917036s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.453469054 (139h29m14.149317797s) [seg_pos={0 0}, upload=122910]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:43.162545170 (139h29m14.858393923s) [upload=122910]: Done uploading segments... Nov 5 07:09:43.162601467 (139h29m14.85845022s) [upload=122910]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 6.016601331s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.208348814s trace: https://jaeger.dev.storj.io/trace/1b48011ff2f3bc78 FAIL (2025-11-05 07:48:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.208377369s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.975470867s trace: https://jaeger.dev.storj.io/trace/75afb5f91d3935bc FAIL (2025-11-05 07:47:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== v 5 07:09:30.093538136 (139h29m1.789386889s) [upload=122911]: Uploading segments... Nov 5 07:09:30.093576248 (139h29m1.789424992s) [upload=122911]: Got next segment. Inline: false Nov 5 07:09:30.093604072 (139h29m1.789452825s) [upload=122911]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 07:09:30.093608600 (139h29m1.789457343s) [upload=122911]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:30.339289624 (139h29m2.035138367s) [seg_pos={0 0}, upload=122911]: Begin upload segment... Nov 5 07:09:30.339349428 (139h29m2.035198181s) [seg_pos={0 0}, upload=122911]: Done begin upload segment. Nov 5 07:09:30.339386408 (139h29m2.035235151s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340339152 (139h29m2.036187905s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340990803 (139h29m2.036839556s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341444226 (139h29m2.03729298s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341919592 (139h29m2.037768375s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342371553 (139h29m2.038220306s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342835706 (139h29m2.038684449s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343494230 (139h29m2.039342973s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343970537 (139h29m2.03981929s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.345438873 (139h29m2.041287626s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.347000497 (139h29m2.04284925s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.348576918 (139h29m2.044425661s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.350325538 (139h29m2.046174281s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.351367712 (139h29m2.047216465s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.354636787 (139h29m2.050485571s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.356330162 (139h29m2.052178915s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357738744 (139h29m2.053587487s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357956329 (139h29m2.053805172s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:31.727744519 (139h29m3.423593262s) [upload=122911]: Next returned nil segment Nov 5 07:09:31.727759807 (139h29m3.42360856s) [upload=122911]: Waiting for error group managing segments... Nov 5 07:09:32.079494147 (139h29m3.77534289s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.084663190 (139h29m3.780511933s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.140346892 (139h29m3.836195645s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.141263718 (139h29m3.837112472s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.325083005 (139h29m4.020931758s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.327962548 (139h29m4.023811301s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.375450033 (139h29m4.071298786s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.421783842 (139h29m4.117632595s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.435408857 (139h29m4.13125761s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.450128345 (139h29m4.145977088s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.464432925 (139h29m4.160281678s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.483329487 (139h29m4.17917824s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.594589918 (139h29m4.290438661s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:10:29.406200389 (139h30m1.102049142s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:145.79.227.13:4820): protocol: rpc: dial tcp 145.79.227.13:4820: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406293496 (139h30m1.102142249s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb, address:145.79.227.16:4920): protocol: rpc: dial tcp 145.79.227.16:4920: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406452028 (139h30m1.102300781s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:145.79.227.20:4440): protocol: rpc: dial tcp 145.79.227.20:4440: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406646058 (139h30m1.102494801s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:145.79.227.39:4047): protocol: rpc: dial tcp 145.79.227.39:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406963362 (139h30m1.102812125s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: rpc: dial tcp 145.79.227.2:4131: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.407007867 (139h30m1.10285662s) [seg_pos={0 0}, upload=122911]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 5 07:10:29.648039207 (139h30m1.34388797s) [upload=122911]: Done uploading segments... Nov 5 07:10:29.648083852 (139h30m1.343932595s) [upload=122911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.97550405s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.296475544s trace: https://jaeger.dev.storj.io/trace/1b48011ff2f3bc78 FAIL (2025-11-05 07:47:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.296504158s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........XX...............................................] duration: 5.619012904s trace: https://jaeger.dev.storj.io/trace/fe05d2f05e7af5f FAIL (2025-11-05 07:47:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 164056 (139h28m44.854012819s) [upload=122910]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:13.414175591 (139h28m45.110024344s) [seg_pos={0 0}, upload=122910]: Begin upload segment... Nov 5 07:09:13.414231787 (139h28m45.110080541s) [seg_pos={0 0}, upload=122910]: Done begin upload segment. Nov 5 07:09:13.414274930 (139h28m45.110123683s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415327825 (139h28m45.111176578s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415861491 (139h28m45.111710244s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416375510 (139h28m45.112224263s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416678197 (139h28m45.11252695s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417009026 (139h28m45.112857779s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417341269 (139h28m45.113190022s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417656319 (139h28m45.113505072s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.418114902 (139h28m45.113963645s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.420954530 (139h28m45.116803273s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.422854929 (139h28m45.118703682s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.424915342 (139h28m45.120764095s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.426819858 (139h28m45.122668611s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.428917523 (139h28m45.124766276s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.430615777 (139h28m45.12646452s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.432400986 (139h28m45.128249739s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:15.183539119 (139h28m46.879387872s) [upload=122910]: Next returned nil segment Nov 5 07:09:15.183553485 (139h28m46.879402239s) [upload=122910]: Waiting for error group managing segments... Nov 5 07:09:15.515060579 (139h28m47.210909332s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.562983674 (139h28m47.258832417s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.584642687 (139h28m47.28049144s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.591557684 (139h28m47.287406427s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.601517589 (139h28m47.297366333s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.688637153 (139h28m47.384485906s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:42.448516803 (139h29m14.144365566s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.448842534 (139h29m14.144691287s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: rpc: dial tcp 145.79.227.13:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449324783 (139h29m14.145173536s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449692342 (139h29m14.145541085s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450039493 (139h29m14.145888236s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450238873 (139h29m14.146087626s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450426109 (139h29m14.146274862s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450710671 (139h29m14.146559434s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: rpc: dial tcp 145.79.227.20:4437: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450894401 (139h29m14.146743154s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: rpc: dial tcp 145.79.227.25:4617: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.451068282 (139h29m14.146917036s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.453469054 (139h29m14.149317797s) [seg_pos={0 0}, upload=122910]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:43.162545170 (139h29m14.858393923s) [upload=122910]: Done uploading segments... Nov 5 07:09:43.162601467 (139h29m14.85845022s) [upload=122910]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 5.619037672s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.160724479s trace: https://jaeger.dev.storj.io/trace/1b48011ff2f3bc78 FAIL (2025-11-05 07:47:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.160754658s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.010398128s trace: https://jaeger.dev.storj.io/trace/75afb5f91d3935bc FAIL (2025-11-05 07:47:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== v 5 07:09:30.093538136 (139h29m1.789386889s) [upload=122911]: Uploading segments... Nov 5 07:09:30.093576248 (139h29m1.789424992s) [upload=122911]: Got next segment. Inline: false Nov 5 07:09:30.093604072 (139h29m1.789452825s) [upload=122911]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 07:09:30.093608600 (139h29m1.789457343s) [upload=122911]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:30.339289624 (139h29m2.035138367s) [seg_pos={0 0}, upload=122911]: Begin upload segment... Nov 5 07:09:30.339349428 (139h29m2.035198181s) [seg_pos={0 0}, upload=122911]: Done begin upload segment. Nov 5 07:09:30.339386408 (139h29m2.035235151s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340339152 (139h29m2.036187905s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340990803 (139h29m2.036839556s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341444226 (139h29m2.03729298s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341919592 (139h29m2.037768375s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342371553 (139h29m2.038220306s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342835706 (139h29m2.038684449s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343494230 (139h29m2.039342973s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343970537 (139h29m2.03981929s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.345438873 (139h29m2.041287626s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.347000497 (139h29m2.04284925s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.348576918 (139h29m2.044425661s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.350325538 (139h29m2.046174281s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.351367712 (139h29m2.047216465s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.354636787 (139h29m2.050485571s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.356330162 (139h29m2.052178915s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357738744 (139h29m2.053587487s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357956329 (139h29m2.053805172s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:31.727744519 (139h29m3.423593262s) [upload=122911]: Next returned nil segment Nov 5 07:09:31.727759807 (139h29m3.42360856s) [upload=122911]: Waiting for error group managing segments... Nov 5 07:09:32.079494147 (139h29m3.77534289s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.084663190 (139h29m3.780511933s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.140346892 (139h29m3.836195645s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.141263718 (139h29m3.837112472s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.325083005 (139h29m4.020931758s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.327962548 (139h29m4.023811301s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.375450033 (139h29m4.071298786s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.421783842 (139h29m4.117632595s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.435408857 (139h29m4.13125761s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.450128345 (139h29m4.145977088s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.464432925 (139h29m4.160281678s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.483329487 (139h29m4.17917824s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.594589918 (139h29m4.290438661s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:10:29.406200389 (139h30m1.102049142s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:145.79.227.13:4820): protocol: rpc: dial tcp 145.79.227.13:4820: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406293496 (139h30m1.102142249s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb, address:145.79.227.16:4920): protocol: rpc: dial tcp 145.79.227.16:4920: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406452028 (139h30m1.102300781s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:145.79.227.20:4440): protocol: rpc: dial tcp 145.79.227.20:4440: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406646058 (139h30m1.102494801s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:145.79.227.39:4047): protocol: rpc: dial tcp 145.79.227.39:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406963362 (139h30m1.102812125s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: rpc: dial tcp 145.79.227.2:4131: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.407007867 (139h30m1.10285662s) [seg_pos={0 0}, upload=122911]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 5 07:10:29.648039207 (139h30m1.34388797s) [upload=122911]: Done uploading segments... Nov 5 07:10:29.648083852 (139h30m1.343932595s) [upload=122911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.010425581s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............XX..............................................] duration: 5.95980107s trace: https://jaeger.dev.storj.io/trace/fe05d2f05e7af5f FAIL (2025-11-05 07:47:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 164056 (139h28m44.854012819s) [upload=122910]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:13.414175591 (139h28m45.110024344s) [seg_pos={0 0}, upload=122910]: Begin upload segment... Nov 5 07:09:13.414231787 (139h28m45.110080541s) [seg_pos={0 0}, upload=122910]: Done begin upload segment. Nov 5 07:09:13.414274930 (139h28m45.110123683s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415327825 (139h28m45.111176578s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415861491 (139h28m45.111710244s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416375510 (139h28m45.112224263s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416678197 (139h28m45.11252695s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417009026 (139h28m45.112857779s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417341269 (139h28m45.113190022s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417656319 (139h28m45.113505072s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.418114902 (139h28m45.113963645s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.420954530 (139h28m45.116803273s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.422854929 (139h28m45.118703682s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.424915342 (139h28m45.120764095s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.426819858 (139h28m45.122668611s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.428917523 (139h28m45.124766276s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.430615777 (139h28m45.12646452s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.432400986 (139h28m45.128249739s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:15.183539119 (139h28m46.879387872s) [upload=122910]: Next returned nil segment Nov 5 07:09:15.183553485 (139h28m46.879402239s) [upload=122910]: Waiting for error group managing segments... Nov 5 07:09:15.515060579 (139h28m47.210909332s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.562983674 (139h28m47.258832417s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.584642687 (139h28m47.28049144s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.591557684 (139h28m47.287406427s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.601517589 (139h28m47.297366333s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.688637153 (139h28m47.384485906s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:42.448516803 (139h29m14.144365566s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.448842534 (139h29m14.144691287s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: rpc: dial tcp 145.79.227.13:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449324783 (139h29m14.145173536s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449692342 (139h29m14.145541085s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450039493 (139h29m14.145888236s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450238873 (139h29m14.146087626s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450426109 (139h29m14.146274862s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450710671 (139h29m14.146559434s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: rpc: dial tcp 145.79.227.20:4437: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450894401 (139h29m14.146743154s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: rpc: dial tcp 145.79.227.25:4617: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.451068282 (139h29m14.146917036s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.453469054 (139h29m14.149317797s) [seg_pos={0 0}, upload=122910]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:43.162545170 (139h29m14.858393923s) [upload=122910]: Done uploading segments... Nov 5 07:09:43.162601467 (139h29m14.85845022s) [upload=122910]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 5.959842871s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.097103663s trace: https://jaeger.dev.storj.io/trace/1b48011ff2f3bc78 FAIL (2025-11-05 07:47:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.097140051s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.342693947s trace: https://jaeger.dev.storj.io/trace/1b48011ff2f3bc78 FAIL (2025-11-05 07:46:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.342760655s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............XX.............................................] duration: 5.942783591s trace: https://jaeger.dev.storj.io/trace/fe05d2f05e7af5f FAIL (2025-11-05 07:46:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 164056 (139h28m44.854012819s) [upload=122910]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:13.414175591 (139h28m45.110024344s) [seg_pos={0 0}, upload=122910]: Begin upload segment... Nov 5 07:09:13.414231787 (139h28m45.110080541s) [seg_pos={0 0}, upload=122910]: Done begin upload segment. Nov 5 07:09:13.414274930 (139h28m45.110123683s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415327825 (139h28m45.111176578s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415861491 (139h28m45.111710244s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416375510 (139h28m45.112224263s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416678197 (139h28m45.11252695s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417009026 (139h28m45.112857779s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417341269 (139h28m45.113190022s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417656319 (139h28m45.113505072s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.418114902 (139h28m45.113963645s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.420954530 (139h28m45.116803273s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.422854929 (139h28m45.118703682s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.424915342 (139h28m45.120764095s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.426819858 (139h28m45.122668611s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.428917523 (139h28m45.124766276s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.430615777 (139h28m45.12646452s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.432400986 (139h28m45.128249739s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:15.183539119 (139h28m46.879387872s) [upload=122910]: Next returned nil segment Nov 5 07:09:15.183553485 (139h28m46.879402239s) [upload=122910]: Waiting for error group managing segments... Nov 5 07:09:15.515060579 (139h28m47.210909332s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.562983674 (139h28m47.258832417s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.584642687 (139h28m47.28049144s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.591557684 (139h28m47.287406427s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.601517589 (139h28m47.297366333s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.688637153 (139h28m47.384485906s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:42.448516803 (139h29m14.144365566s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.448842534 (139h29m14.144691287s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: rpc: dial tcp 145.79.227.13:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449324783 (139h29m14.145173536s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449692342 (139h29m14.145541085s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450039493 (139h29m14.145888236s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450238873 (139h29m14.146087626s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450426109 (139h29m14.146274862s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450710671 (139h29m14.146559434s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: rpc: dial tcp 145.79.227.20:4437: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450894401 (139h29m14.146743154s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: rpc: dial tcp 145.79.227.25:4617: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.451068282 (139h29m14.146917036s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.453469054 (139h29m14.149317797s) [seg_pos={0 0}, upload=122910]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:43.162545170 (139h29m14.858393923s) [upload=122910]: Done uploading segments... Nov 5 07:09:43.162601467 (139h29m14.85845022s) [upload=122910]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 5.942812888s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.919622126s trace: https://jaeger.dev.storj.io/trace/75afb5f91d3935bc FAIL (2025-11-05 07:46:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== v 5 07:09:30.093538136 (139h29m1.789386889s) [upload=122911]: Uploading segments... Nov 5 07:09:30.093576248 (139h29m1.789424992s) [upload=122911]: Got next segment. Inline: false Nov 5 07:09:30.093604072 (139h29m1.789452825s) [upload=122911]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 07:09:30.093608600 (139h29m1.789457343s) [upload=122911]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:30.339289624 (139h29m2.035138367s) [seg_pos={0 0}, upload=122911]: Begin upload segment... Nov 5 07:09:30.339349428 (139h29m2.035198181s) [seg_pos={0 0}, upload=122911]: Done begin upload segment. Nov 5 07:09:30.339386408 (139h29m2.035235151s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340339152 (139h29m2.036187905s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340990803 (139h29m2.036839556s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341444226 (139h29m2.03729298s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341919592 (139h29m2.037768375s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342371553 (139h29m2.038220306s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342835706 (139h29m2.038684449s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343494230 (139h29m2.039342973s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343970537 (139h29m2.03981929s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.345438873 (139h29m2.041287626s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.347000497 (139h29m2.04284925s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.348576918 (139h29m2.044425661s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.350325538 (139h29m2.046174281s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.351367712 (139h29m2.047216465s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.354636787 (139h29m2.050485571s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.356330162 (139h29m2.052178915s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357738744 (139h29m2.053587487s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357956329 (139h29m2.053805172s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:31.727744519 (139h29m3.423593262s) [upload=122911]: Next returned nil segment Nov 5 07:09:31.727759807 (139h29m3.42360856s) [upload=122911]: Waiting for error group managing segments... Nov 5 07:09:32.079494147 (139h29m3.77534289s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.084663190 (139h29m3.780511933s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.140346892 (139h29m3.836195645s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.141263718 (139h29m3.837112472s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.325083005 (139h29m4.020931758s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.327962548 (139h29m4.023811301s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.375450033 (139h29m4.071298786s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.421783842 (139h29m4.117632595s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.435408857 (139h29m4.13125761s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.450128345 (139h29m4.145977088s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.464432925 (139h29m4.160281678s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.483329487 (139h29m4.17917824s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.594589918 (139h29m4.290438661s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:10:29.406200389 (139h30m1.102049142s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:145.79.227.13:4820): protocol: rpc: dial tcp 145.79.227.13:4820: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406293496 (139h30m1.102142249s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb, address:145.79.227.16:4920): protocol: rpc: dial tcp 145.79.227.16:4920: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406452028 (139h30m1.102300781s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:145.79.227.20:4440): protocol: rpc: dial tcp 145.79.227.20:4440: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406646058 (139h30m1.102494801s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:145.79.227.39:4047): protocol: rpc: dial tcp 145.79.227.39:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406963362 (139h30m1.102812125s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: rpc: dial tcp 145.79.227.2:4131: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.407007867 (139h30m1.10285662s) [seg_pos={0 0}, upload=122911]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 5 07:10:29.648039207 (139h30m1.34388797s) [upload=122911]: Done uploading segments... Nov 5 07:10:29.648083852 (139h30m1.343932595s) [upload=122911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.919650932s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.909356124s trace: https://jaeger.dev.storj.io/trace/75afb5f91d3935bc FAIL (2025-11-05 07:45:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== v 5 07:09:30.093538136 (139h29m1.789386889s) [upload=122911]: Uploading segments... Nov 5 07:09:30.093576248 (139h29m1.789424992s) [upload=122911]: Got next segment. Inline: false Nov 5 07:09:30.093604072 (139h29m1.789452825s) [upload=122911]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 07:09:30.093608600 (139h29m1.789457343s) [upload=122911]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:30.339289624 (139h29m2.035138367s) [seg_pos={0 0}, upload=122911]: Begin upload segment... Nov 5 07:09:30.339349428 (139h29m2.035198181s) [seg_pos={0 0}, upload=122911]: Done begin upload segment. Nov 5 07:09:30.339386408 (139h29m2.035235151s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340339152 (139h29m2.036187905s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340990803 (139h29m2.036839556s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341444226 (139h29m2.03729298s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341919592 (139h29m2.037768375s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342371553 (139h29m2.038220306s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342835706 (139h29m2.038684449s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343494230 (139h29m2.039342973s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343970537 (139h29m2.03981929s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.345438873 (139h29m2.041287626s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.347000497 (139h29m2.04284925s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.348576918 (139h29m2.044425661s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.350325538 (139h29m2.046174281s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.351367712 (139h29m2.047216465s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.354636787 (139h29m2.050485571s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.356330162 (139h29m2.052178915s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357738744 (139h29m2.053587487s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357956329 (139h29m2.053805172s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:31.727744519 (139h29m3.423593262s) [upload=122911]: Next returned nil segment Nov 5 07:09:31.727759807 (139h29m3.42360856s) [upload=122911]: Waiting for error group managing segments... Nov 5 07:09:32.079494147 (139h29m3.77534289s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.084663190 (139h29m3.780511933s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.140346892 (139h29m3.836195645s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.141263718 (139h29m3.837112472s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.325083005 (139h29m4.020931758s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.327962548 (139h29m4.023811301s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.375450033 (139h29m4.071298786s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.421783842 (139h29m4.117632595s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.435408857 (139h29m4.13125761s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.450128345 (139h29m4.145977088s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.464432925 (139h29m4.160281678s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.483329487 (139h29m4.17917824s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.594589918 (139h29m4.290438661s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:10:29.406200389 (139h30m1.102049142s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:145.79.227.13:4820): protocol: rpc: dial tcp 145.79.227.13:4820: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406293496 (139h30m1.102142249s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb, address:145.79.227.16:4920): protocol: rpc: dial tcp 145.79.227.16:4920: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406452028 (139h30m1.102300781s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:145.79.227.20:4440): protocol: rpc: dial tcp 145.79.227.20:4440: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406646058 (139h30m1.102494801s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:145.79.227.39:4047): protocol: rpc: dial tcp 145.79.227.39:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406963362 (139h30m1.102812125s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: rpc: dial tcp 145.79.227.2:4131: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.407007867 (139h30m1.10285662s) [seg_pos={0 0}, upload=122911]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 5 07:10:29.648039207 (139h30m1.34388797s) [upload=122911]: Done uploading segments... Nov 5 07:10:29.648083852 (139h30m1.343932595s) [upload=122911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.909384849s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............XX............................................] duration: 5.718696506s trace: https://jaeger.dev.storj.io/trace/fe05d2f05e7af5f FAIL (2025-11-05 07:45:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 164056 (139h28m44.854012819s) [upload=122910]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:13.414175591 (139h28m45.110024344s) [seg_pos={0 0}, upload=122910]: Begin upload segment... Nov 5 07:09:13.414231787 (139h28m45.110080541s) [seg_pos={0 0}, upload=122910]: Done begin upload segment. Nov 5 07:09:13.414274930 (139h28m45.110123683s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415327825 (139h28m45.111176578s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415861491 (139h28m45.111710244s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416375510 (139h28m45.112224263s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416678197 (139h28m45.11252695s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417009026 (139h28m45.112857779s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417341269 (139h28m45.113190022s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417656319 (139h28m45.113505072s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.418114902 (139h28m45.113963645s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.420954530 (139h28m45.116803273s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.422854929 (139h28m45.118703682s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.424915342 (139h28m45.120764095s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.426819858 (139h28m45.122668611s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.428917523 (139h28m45.124766276s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.430615777 (139h28m45.12646452s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.432400986 (139h28m45.128249739s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:15.183539119 (139h28m46.879387872s) [upload=122910]: Next returned nil segment Nov 5 07:09:15.183553485 (139h28m46.879402239s) [upload=122910]: Waiting for error group managing segments... Nov 5 07:09:15.515060579 (139h28m47.210909332s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.562983674 (139h28m47.258832417s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.584642687 (139h28m47.28049144s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.591557684 (139h28m47.287406427s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.601517589 (139h28m47.297366333s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.688637153 (139h28m47.384485906s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:42.448516803 (139h29m14.144365566s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.448842534 (139h29m14.144691287s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: rpc: dial tcp 145.79.227.13:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449324783 (139h29m14.145173536s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449692342 (139h29m14.145541085s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450039493 (139h29m14.145888236s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450238873 (139h29m14.146087626s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450426109 (139h29m14.146274862s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450710671 (139h29m14.146559434s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: rpc: dial tcp 145.79.227.20:4437: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450894401 (139h29m14.146743154s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: rpc: dial tcp 145.79.227.25:4617: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.451068282 (139h29m14.146917036s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.453469054 (139h29m14.149317797s) [seg_pos={0 0}, upload=122910]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:43.162545170 (139h29m14.858393923s) [upload=122910]: Done uploading segments... Nov 5 07:09:43.162601467 (139h29m14.85845022s) [upload=122910]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 5.718727276s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.27502276s trace: https://jaeger.dev.storj.io/trace/1b48011ff2f3bc78 FAIL (2025-11-05 07:45:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.275079227s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.402381962s trace: https://jaeger.dev.storj.io/trace/1b48011ff2f3bc78 FAIL (2025-11-05 07:44:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.402405066s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.046669284s trace: https://jaeger.dev.storj.io/trace/75afb5f91d3935bc FAIL (2025-11-05 07:44:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== v 5 07:09:30.093538136 (139h29m1.789386889s) [upload=122911]: Uploading segments... Nov 5 07:09:30.093576248 (139h29m1.789424992s) [upload=122911]: Got next segment. Inline: false Nov 5 07:09:30.093604072 (139h29m1.789452825s) [upload=122911]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 07:09:30.093608600 (139h29m1.789457343s) [upload=122911]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:30.339289624 (139h29m2.035138367s) [seg_pos={0 0}, upload=122911]: Begin upload segment... Nov 5 07:09:30.339349428 (139h29m2.035198181s) [seg_pos={0 0}, upload=122911]: Done begin upload segment. Nov 5 07:09:30.339386408 (139h29m2.035235151s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340339152 (139h29m2.036187905s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340990803 (139h29m2.036839556s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341444226 (139h29m2.03729298s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341919592 (139h29m2.037768375s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342371553 (139h29m2.038220306s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342835706 (139h29m2.038684449s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343494230 (139h29m2.039342973s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343970537 (139h29m2.03981929s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.345438873 (139h29m2.041287626s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.347000497 (139h29m2.04284925s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.348576918 (139h29m2.044425661s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.350325538 (139h29m2.046174281s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.351367712 (139h29m2.047216465s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.354636787 (139h29m2.050485571s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.356330162 (139h29m2.052178915s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357738744 (139h29m2.053587487s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357956329 (139h29m2.053805172s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:31.727744519 (139h29m3.423593262s) [upload=122911]: Next returned nil segment Nov 5 07:09:31.727759807 (139h29m3.42360856s) [upload=122911]: Waiting for error group managing segments... Nov 5 07:09:32.079494147 (139h29m3.77534289s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.084663190 (139h29m3.780511933s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.140346892 (139h29m3.836195645s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.141263718 (139h29m3.837112472s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.325083005 (139h29m4.020931758s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.327962548 (139h29m4.023811301s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.375450033 (139h29m4.071298786s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.421783842 (139h29m4.117632595s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.435408857 (139h29m4.13125761s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.450128345 (139h29m4.145977088s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.464432925 (139h29m4.160281678s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.483329487 (139h29m4.17917824s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.594589918 (139h29m4.290438661s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:10:29.406200389 (139h30m1.102049142s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:145.79.227.13:4820): protocol: rpc: dial tcp 145.79.227.13:4820: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406293496 (139h30m1.102142249s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb, address:145.79.227.16:4920): protocol: rpc: dial tcp 145.79.227.16:4920: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406452028 (139h30m1.102300781s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:145.79.227.20:4440): protocol: rpc: dial tcp 145.79.227.20:4440: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406646058 (139h30m1.102494801s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:145.79.227.39:4047): protocol: rpc: dial tcp 145.79.227.39:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406963362 (139h30m1.102812125s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: rpc: dial tcp 145.79.227.2:4131: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.407007867 (139h30m1.10285662s) [seg_pos={0 0}, upload=122911]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 5 07:10:29.648039207 (139h30m1.34388797s) [upload=122911]: Done uploading segments... Nov 5 07:10:29.648083852 (139h30m1.343932595s) [upload=122911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.046708219s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............XX...........................................] duration: 5.84322093s trace: https://jaeger.dev.storj.io/trace/fe05d2f05e7af5f FAIL (2025-11-05 07:44:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 164056 (139h28m44.854012819s) [upload=122910]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:13.414175591 (139h28m45.110024344s) [seg_pos={0 0}, upload=122910]: Begin upload segment... Nov 5 07:09:13.414231787 (139h28m45.110080541s) [seg_pos={0 0}, upload=122910]: Done begin upload segment. Nov 5 07:09:13.414274930 (139h28m45.110123683s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415327825 (139h28m45.111176578s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415861491 (139h28m45.111710244s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416375510 (139h28m45.112224263s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416678197 (139h28m45.11252695s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417009026 (139h28m45.112857779s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417341269 (139h28m45.113190022s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417656319 (139h28m45.113505072s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.418114902 (139h28m45.113963645s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.420954530 (139h28m45.116803273s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.422854929 (139h28m45.118703682s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.424915342 (139h28m45.120764095s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.426819858 (139h28m45.122668611s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.428917523 (139h28m45.124766276s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.430615777 (139h28m45.12646452s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.432400986 (139h28m45.128249739s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:15.183539119 (139h28m46.879387872s) [upload=122910]: Next returned nil segment Nov 5 07:09:15.183553485 (139h28m46.879402239s) [upload=122910]: Waiting for error group managing segments... Nov 5 07:09:15.515060579 (139h28m47.210909332s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.562983674 (139h28m47.258832417s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.584642687 (139h28m47.28049144s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.591557684 (139h28m47.287406427s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.601517589 (139h28m47.297366333s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.688637153 (139h28m47.384485906s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:42.448516803 (139h29m14.144365566s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.448842534 (139h29m14.144691287s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: rpc: dial tcp 145.79.227.13:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449324783 (139h29m14.145173536s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449692342 (139h29m14.145541085s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450039493 (139h29m14.145888236s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450238873 (139h29m14.146087626s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450426109 (139h29m14.146274862s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450710671 (139h29m14.146559434s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: rpc: dial tcp 145.79.227.20:4437: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450894401 (139h29m14.146743154s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: rpc: dial tcp 145.79.227.25:4617: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.451068282 (139h29m14.146917036s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.453469054 (139h29m14.149317797s) [seg_pos={0 0}, upload=122910]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:43.162545170 (139h29m14.858393923s) [upload=122910]: Done uploading segments... Nov 5 07:09:43.162601467 (139h29m14.85845022s) [upload=122910]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 5.843259332s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.412617846s trace: https://jaeger.dev.storj.io/trace/1b48011ff2f3bc78 FAIL (2025-11-05 07:44:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.412747081s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................XX..........................................] duration: 5.796204425s trace: https://jaeger.dev.storj.io/trace/fe05d2f05e7af5f FAIL (2025-11-05 07:44:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 164056 (139h28m44.854012819s) [upload=122910]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:13.414175591 (139h28m45.110024344s) [seg_pos={0 0}, upload=122910]: Begin upload segment... Nov 5 07:09:13.414231787 (139h28m45.110080541s) [seg_pos={0 0}, upload=122910]: Done begin upload segment. Nov 5 07:09:13.414274930 (139h28m45.110123683s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415327825 (139h28m45.111176578s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415861491 (139h28m45.111710244s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416375510 (139h28m45.112224263s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416678197 (139h28m45.11252695s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417009026 (139h28m45.112857779s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417341269 (139h28m45.113190022s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417656319 (139h28m45.113505072s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.418114902 (139h28m45.113963645s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.420954530 (139h28m45.116803273s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.422854929 (139h28m45.118703682s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.424915342 (139h28m45.120764095s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.426819858 (139h28m45.122668611s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.428917523 (139h28m45.124766276s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.430615777 (139h28m45.12646452s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.432400986 (139h28m45.128249739s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:15.183539119 (139h28m46.879387872s) [upload=122910]: Next returned nil segment Nov 5 07:09:15.183553485 (139h28m46.879402239s) [upload=122910]: Waiting for error group managing segments... Nov 5 07:09:15.515060579 (139h28m47.210909332s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.562983674 (139h28m47.258832417s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.584642687 (139h28m47.28049144s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.591557684 (139h28m47.287406427s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.601517589 (139h28m47.297366333s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.688637153 (139h28m47.384485906s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:42.448516803 (139h29m14.144365566s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.448842534 (139h29m14.144691287s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: rpc: dial tcp 145.79.227.13:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449324783 (139h29m14.145173536s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449692342 (139h29m14.145541085s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450039493 (139h29m14.145888236s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450238873 (139h29m14.146087626s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450426109 (139h29m14.146274862s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450710671 (139h29m14.146559434s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: rpc: dial tcp 145.79.227.20:4437: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450894401 (139h29m14.146743154s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: rpc: dial tcp 145.79.227.25:4617: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.451068282 (139h29m14.146917036s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.453469054 (139h29m14.149317797s) [seg_pos={0 0}, upload=122910]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:43.162545170 (139h29m14.858393923s) [upload=122910]: Done uploading segments... Nov 5 07:09:43.162601467 (139h29m14.85845022s) [upload=122910]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 5.796248758s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.878057559s trace: https://jaeger.dev.storj.io/trace/75afb5f91d3935bc FAIL (2025-11-05 07:43:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== v 5 07:09:30.093538136 (139h29m1.789386889s) [upload=122911]: Uploading segments... Nov 5 07:09:30.093576248 (139h29m1.789424992s) [upload=122911]: Got next segment. Inline: false Nov 5 07:09:30.093604072 (139h29m1.789452825s) [upload=122911]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 07:09:30.093608600 (139h29m1.789457343s) [upload=122911]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:30.339289624 (139h29m2.035138367s) [seg_pos={0 0}, upload=122911]: Begin upload segment... Nov 5 07:09:30.339349428 (139h29m2.035198181s) [seg_pos={0 0}, upload=122911]: Done begin upload segment. Nov 5 07:09:30.339386408 (139h29m2.035235151s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340339152 (139h29m2.036187905s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340990803 (139h29m2.036839556s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341444226 (139h29m2.03729298s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341919592 (139h29m2.037768375s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342371553 (139h29m2.038220306s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342835706 (139h29m2.038684449s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343494230 (139h29m2.039342973s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343970537 (139h29m2.03981929s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.345438873 (139h29m2.041287626s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.347000497 (139h29m2.04284925s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.348576918 (139h29m2.044425661s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.350325538 (139h29m2.046174281s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.351367712 (139h29m2.047216465s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.354636787 (139h29m2.050485571s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.356330162 (139h29m2.052178915s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357738744 (139h29m2.053587487s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357956329 (139h29m2.053805172s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:31.727744519 (139h29m3.423593262s) [upload=122911]: Next returned nil segment Nov 5 07:09:31.727759807 (139h29m3.42360856s) [upload=122911]: Waiting for error group managing segments... Nov 5 07:09:32.079494147 (139h29m3.77534289s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.084663190 (139h29m3.780511933s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.140346892 (139h29m3.836195645s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.141263718 (139h29m3.837112472s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.325083005 (139h29m4.020931758s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.327962548 (139h29m4.023811301s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.375450033 (139h29m4.071298786s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.421783842 (139h29m4.117632595s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.435408857 (139h29m4.13125761s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.450128345 (139h29m4.145977088s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.464432925 (139h29m4.160281678s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.483329487 (139h29m4.17917824s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.594589918 (139h29m4.290438661s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:10:29.406200389 (139h30m1.102049142s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:145.79.227.13:4820): protocol: rpc: dial tcp 145.79.227.13:4820: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406293496 (139h30m1.102142249s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb, address:145.79.227.16:4920): protocol: rpc: dial tcp 145.79.227.16:4920: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406452028 (139h30m1.102300781s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:145.79.227.20:4440): protocol: rpc: dial tcp 145.79.227.20:4440: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406646058 (139h30m1.102494801s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:145.79.227.39:4047): protocol: rpc: dial tcp 145.79.227.39:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406963362 (139h30m1.102812125s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: rpc: dial tcp 145.79.227.2:4131: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.407007867 (139h30m1.10285662s) [seg_pos={0 0}, upload=122911]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 5 07:10:29.648039207 (139h30m1.34388797s) [upload=122911]: Done uploading segments... Nov 5 07:10:29.648083852 (139h30m1.343932595s) [upload=122911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.878086343s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.150885651s trace: https://jaeger.dev.storj.io/trace/1b48011ff2f3bc78 FAIL (2025-11-05 07:43:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.150935014s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................XX.........................................] duration: 5.758807083s trace: https://jaeger.dev.storj.io/trace/fe05d2f05e7af5f FAIL (2025-11-05 07:43:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 164056 (139h28m44.854012819s) [upload=122910]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:13.414175591 (139h28m45.110024344s) [seg_pos={0 0}, upload=122910]: Begin upload segment... Nov 5 07:09:13.414231787 (139h28m45.110080541s) [seg_pos={0 0}, upload=122910]: Done begin upload segment. Nov 5 07:09:13.414274930 (139h28m45.110123683s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415327825 (139h28m45.111176578s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415861491 (139h28m45.111710244s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416375510 (139h28m45.112224263s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416678197 (139h28m45.11252695s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417009026 (139h28m45.112857779s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417341269 (139h28m45.113190022s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417656319 (139h28m45.113505072s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.418114902 (139h28m45.113963645s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.420954530 (139h28m45.116803273s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.422854929 (139h28m45.118703682s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.424915342 (139h28m45.120764095s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.426819858 (139h28m45.122668611s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.428917523 (139h28m45.124766276s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.430615777 (139h28m45.12646452s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.432400986 (139h28m45.128249739s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:15.183539119 (139h28m46.879387872s) [upload=122910]: Next returned nil segment Nov 5 07:09:15.183553485 (139h28m46.879402239s) [upload=122910]: Waiting for error group managing segments... Nov 5 07:09:15.515060579 (139h28m47.210909332s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.562983674 (139h28m47.258832417s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.584642687 (139h28m47.28049144s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.591557684 (139h28m47.287406427s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.601517589 (139h28m47.297366333s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.688637153 (139h28m47.384485906s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:42.448516803 (139h29m14.144365566s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.448842534 (139h29m14.144691287s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: rpc: dial tcp 145.79.227.13:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449324783 (139h29m14.145173536s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449692342 (139h29m14.145541085s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450039493 (139h29m14.145888236s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450238873 (139h29m14.146087626s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450426109 (139h29m14.146274862s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450710671 (139h29m14.146559434s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: rpc: dial tcp 145.79.227.20:4437: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450894401 (139h29m14.146743154s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: rpc: dial tcp 145.79.227.25:4617: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.451068282 (139h29m14.146917036s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.453469054 (139h29m14.149317797s) [seg_pos={0 0}, upload=122910]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:43.162545170 (139h29m14.858393923s) [upload=122910]: Done uploading segments... Nov 5 07:09:43.162601467 (139h29m14.85845022s) [upload=122910]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 5.7588363s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.144483515s trace: https://jaeger.dev.storj.io/trace/1b48011ff2f3bc78 FAIL (2025-11-05 07:43:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.14451295s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................XX........................................] duration: 5.913584441s trace: https://jaeger.dev.storj.io/trace/fe05d2f05e7af5f FAIL (2025-11-05 07:43:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 164056 (139h28m44.854012819s) [upload=122910]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:13.414175591 (139h28m45.110024344s) [seg_pos={0 0}, upload=122910]: Begin upload segment... Nov 5 07:09:13.414231787 (139h28m45.110080541s) [seg_pos={0 0}, upload=122910]: Done begin upload segment. Nov 5 07:09:13.414274930 (139h28m45.110123683s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415327825 (139h28m45.111176578s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415861491 (139h28m45.111710244s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416375510 (139h28m45.112224263s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416678197 (139h28m45.11252695s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417009026 (139h28m45.112857779s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417341269 (139h28m45.113190022s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417656319 (139h28m45.113505072s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.418114902 (139h28m45.113963645s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.420954530 (139h28m45.116803273s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.422854929 (139h28m45.118703682s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.424915342 (139h28m45.120764095s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.426819858 (139h28m45.122668611s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.428917523 (139h28m45.124766276s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.430615777 (139h28m45.12646452s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.432400986 (139h28m45.128249739s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:15.183539119 (139h28m46.879387872s) [upload=122910]: Next returned nil segment Nov 5 07:09:15.183553485 (139h28m46.879402239s) [upload=122910]: Waiting for error group managing segments... Nov 5 07:09:15.515060579 (139h28m47.210909332s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.562983674 (139h28m47.258832417s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.584642687 (139h28m47.28049144s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.591557684 (139h28m47.287406427s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.601517589 (139h28m47.297366333s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.688637153 (139h28m47.384485906s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:42.448516803 (139h29m14.144365566s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.448842534 (139h29m14.144691287s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: rpc: dial tcp 145.79.227.13:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449324783 (139h29m14.145173536s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449692342 (139h29m14.145541085s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450039493 (139h29m14.145888236s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450238873 (139h29m14.146087626s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450426109 (139h29m14.146274862s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450710671 (139h29m14.146559434s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: rpc: dial tcp 145.79.227.20:4437: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450894401 (139h29m14.146743154s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: rpc: dial tcp 145.79.227.25:4617: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.451068282 (139h29m14.146917036s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.453469054 (139h29m14.149317797s) [seg_pos={0 0}, upload=122910]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:43.162545170 (139h29m14.858393923s) [upload=122910]: Done uploading segments... Nov 5 07:09:43.162601467 (139h29m14.85845022s) [upload=122910]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 5.913619037s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.109152806s trace: https://jaeger.dev.storj.io/trace/75afb5f91d3935bc FAIL (2025-11-05 07:43:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== v 5 07:09:30.093538136 (139h29m1.789386889s) [upload=122911]: Uploading segments... Nov 5 07:09:30.093576248 (139h29m1.789424992s) [upload=122911]: Got next segment. Inline: false Nov 5 07:09:30.093604072 (139h29m1.789452825s) [upload=122911]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 07:09:30.093608600 (139h29m1.789457343s) [upload=122911]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:30.339289624 (139h29m2.035138367s) [seg_pos={0 0}, upload=122911]: Begin upload segment... Nov 5 07:09:30.339349428 (139h29m2.035198181s) [seg_pos={0 0}, upload=122911]: Done begin upload segment. Nov 5 07:09:30.339386408 (139h29m2.035235151s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340339152 (139h29m2.036187905s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340990803 (139h29m2.036839556s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341444226 (139h29m2.03729298s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341919592 (139h29m2.037768375s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342371553 (139h29m2.038220306s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342835706 (139h29m2.038684449s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343494230 (139h29m2.039342973s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343970537 (139h29m2.03981929s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.345438873 (139h29m2.041287626s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.347000497 (139h29m2.04284925s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.348576918 (139h29m2.044425661s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.350325538 (139h29m2.046174281s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.351367712 (139h29m2.047216465s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.354636787 (139h29m2.050485571s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.356330162 (139h29m2.052178915s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357738744 (139h29m2.053587487s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357956329 (139h29m2.053805172s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:31.727744519 (139h29m3.423593262s) [upload=122911]: Next returned nil segment Nov 5 07:09:31.727759807 (139h29m3.42360856s) [upload=122911]: Waiting for error group managing segments... Nov 5 07:09:32.079494147 (139h29m3.77534289s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.084663190 (139h29m3.780511933s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.140346892 (139h29m3.836195645s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.141263718 (139h29m3.837112472s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.325083005 (139h29m4.020931758s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.327962548 (139h29m4.023811301s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.375450033 (139h29m4.071298786s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.421783842 (139h29m4.117632595s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.435408857 (139h29m4.13125761s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.450128345 (139h29m4.145977088s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.464432925 (139h29m4.160281678s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.483329487 (139h29m4.17917824s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.594589918 (139h29m4.290438661s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:10:29.406200389 (139h30m1.102049142s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:145.79.227.13:4820): protocol: rpc: dial tcp 145.79.227.13:4820: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406293496 (139h30m1.102142249s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb, address:145.79.227.16:4920): protocol: rpc: dial tcp 145.79.227.16:4920: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406452028 (139h30m1.102300781s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:145.79.227.20:4440): protocol: rpc: dial tcp 145.79.227.20:4440: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406646058 (139h30m1.102494801s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:145.79.227.39:4047): protocol: rpc: dial tcp 145.79.227.39:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406963362 (139h30m1.102812125s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: rpc: dial tcp 145.79.227.2:4131: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.407007867 (139h30m1.10285662s) [seg_pos={0 0}, upload=122911]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 5 07:10:29.648039207 (139h30m1.34388797s) [upload=122911]: Done uploading segments... Nov 5 07:10:29.648083852 (139h30m1.343932595s) [upload=122911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.10918617s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.09157229s trace: https://jaeger.dev.storj.io/trace/1b48011ff2f3bc78 FAIL (2025-11-05 07:42:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.091605784s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.20627091s trace: https://jaeger.dev.storj.io/trace/1b48011ff2f3bc78 FAIL (2025-11-05 07:42:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.206306157s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................XX.......................................] duration: 5.865890328s trace: https://jaeger.dev.storj.io/trace/fe05d2f05e7af5f FAIL (2025-11-05 07:42:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 164056 (139h28m44.854012819s) [upload=122910]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:13.414175591 (139h28m45.110024344s) [seg_pos={0 0}, upload=122910]: Begin upload segment... Nov 5 07:09:13.414231787 (139h28m45.110080541s) [seg_pos={0 0}, upload=122910]: Done begin upload segment. Nov 5 07:09:13.414274930 (139h28m45.110123683s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415327825 (139h28m45.111176578s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415861491 (139h28m45.111710244s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416375510 (139h28m45.112224263s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416678197 (139h28m45.11252695s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417009026 (139h28m45.112857779s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417341269 (139h28m45.113190022s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417656319 (139h28m45.113505072s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.418114902 (139h28m45.113963645s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.420954530 (139h28m45.116803273s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.422854929 (139h28m45.118703682s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.424915342 (139h28m45.120764095s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.426819858 (139h28m45.122668611s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.428917523 (139h28m45.124766276s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.430615777 (139h28m45.12646452s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.432400986 (139h28m45.128249739s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:15.183539119 (139h28m46.879387872s) [upload=122910]: Next returned nil segment Nov 5 07:09:15.183553485 (139h28m46.879402239s) [upload=122910]: Waiting for error group managing segments... Nov 5 07:09:15.515060579 (139h28m47.210909332s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.562983674 (139h28m47.258832417s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.584642687 (139h28m47.28049144s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.591557684 (139h28m47.287406427s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.601517589 (139h28m47.297366333s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.688637153 (139h28m47.384485906s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:42.448516803 (139h29m14.144365566s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.448842534 (139h29m14.144691287s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: rpc: dial tcp 145.79.227.13:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449324783 (139h29m14.145173536s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449692342 (139h29m14.145541085s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450039493 (139h29m14.145888236s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450238873 (139h29m14.146087626s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450426109 (139h29m14.146274862s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450710671 (139h29m14.146559434s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: rpc: dial tcp 145.79.227.20:4437: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450894401 (139h29m14.146743154s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: rpc: dial tcp 145.79.227.25:4617: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.451068282 (139h29m14.146917036s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.453469054 (139h29m14.149317797s) [seg_pos={0 0}, upload=122910]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:43.162545170 (139h29m14.858393923s) [upload=122910]: Done uploading segments... Nov 5 07:09:43.162601467 (139h29m14.85845022s) [upload=122910]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 5.865924212s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.887611521s trace: https://jaeger.dev.storj.io/trace/75afb5f91d3935bc FAIL (2025-11-05 07:41:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== v 5 07:09:30.093538136 (139h29m1.789386889s) [upload=122911]: Uploading segments... Nov 5 07:09:30.093576248 (139h29m1.789424992s) [upload=122911]: Got next segment. Inline: false Nov 5 07:09:30.093604072 (139h29m1.789452825s) [upload=122911]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 07:09:30.093608600 (139h29m1.789457343s) [upload=122911]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:30.339289624 (139h29m2.035138367s) [seg_pos={0 0}, upload=122911]: Begin upload segment... Nov 5 07:09:30.339349428 (139h29m2.035198181s) [seg_pos={0 0}, upload=122911]: Done begin upload segment. Nov 5 07:09:30.339386408 (139h29m2.035235151s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340339152 (139h29m2.036187905s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340990803 (139h29m2.036839556s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341444226 (139h29m2.03729298s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341919592 (139h29m2.037768375s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342371553 (139h29m2.038220306s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342835706 (139h29m2.038684449s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343494230 (139h29m2.039342973s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343970537 (139h29m2.03981929s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.345438873 (139h29m2.041287626s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.347000497 (139h29m2.04284925s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.348576918 (139h29m2.044425661s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.350325538 (139h29m2.046174281s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.351367712 (139h29m2.047216465s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.354636787 (139h29m2.050485571s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.356330162 (139h29m2.052178915s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357738744 (139h29m2.053587487s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357956329 (139h29m2.053805172s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:31.727744519 (139h29m3.423593262s) [upload=122911]: Next returned nil segment Nov 5 07:09:31.727759807 (139h29m3.42360856s) [upload=122911]: Waiting for error group managing segments... Nov 5 07:09:32.079494147 (139h29m3.77534289s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.084663190 (139h29m3.780511933s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.140346892 (139h29m3.836195645s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.141263718 (139h29m3.837112472s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.325083005 (139h29m4.020931758s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.327962548 (139h29m4.023811301s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.375450033 (139h29m4.071298786s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.421783842 (139h29m4.117632595s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.435408857 (139h29m4.13125761s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.450128345 (139h29m4.145977088s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.464432925 (139h29m4.160281678s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.483329487 (139h29m4.17917824s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.594589918 (139h29m4.290438661s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:10:29.406200389 (139h30m1.102049142s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:145.79.227.13:4820): protocol: rpc: dial tcp 145.79.227.13:4820: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406293496 (139h30m1.102142249s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb, address:145.79.227.16:4920): protocol: rpc: dial tcp 145.79.227.16:4920: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406452028 (139h30m1.102300781s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:145.79.227.20:4440): protocol: rpc: dial tcp 145.79.227.20:4440: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406646058 (139h30m1.102494801s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:145.79.227.39:4047): protocol: rpc: dial tcp 145.79.227.39:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406963362 (139h30m1.102812125s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: rpc: dial tcp 145.79.227.2:4131: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.407007867 (139h30m1.10285662s) [seg_pos={0 0}, upload=122911]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 5 07:10:29.648039207 (139h30m1.34388797s) [upload=122911]: Done uploading segments... Nov 5 07:10:29.648083852 (139h30m1.343932595s) [upload=122911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.887646487s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.149986591s trace: https://jaeger.dev.storj.io/trace/1b48011ff2f3bc78 FAIL (2025-11-05 07:41:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.150016838s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.168689289s trace: https://jaeger.dev.storj.io/trace/1b48011ff2f3bc78 FAIL (2025-11-05 07:41:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.16872178s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................XX......................................] duration: 5.945421573s trace: https://jaeger.dev.storj.io/trace/fe05d2f05e7af5f FAIL (2025-11-05 07:40:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 164056 (139h28m44.854012819s) [upload=122910]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:13.414175591 (139h28m45.110024344s) [seg_pos={0 0}, upload=122910]: Begin upload segment... Nov 5 07:09:13.414231787 (139h28m45.110080541s) [seg_pos={0 0}, upload=122910]: Done begin upload segment. Nov 5 07:09:13.414274930 (139h28m45.110123683s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415327825 (139h28m45.111176578s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415861491 (139h28m45.111710244s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416375510 (139h28m45.112224263s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416678197 (139h28m45.11252695s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417009026 (139h28m45.112857779s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417341269 (139h28m45.113190022s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417656319 (139h28m45.113505072s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.418114902 (139h28m45.113963645s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.420954530 (139h28m45.116803273s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.422854929 (139h28m45.118703682s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.424915342 (139h28m45.120764095s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.426819858 (139h28m45.122668611s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.428917523 (139h28m45.124766276s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.430615777 (139h28m45.12646452s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.432400986 (139h28m45.128249739s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:15.183539119 (139h28m46.879387872s) [upload=122910]: Next returned nil segment Nov 5 07:09:15.183553485 (139h28m46.879402239s) [upload=122910]: Waiting for error group managing segments... Nov 5 07:09:15.515060579 (139h28m47.210909332s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.562983674 (139h28m47.258832417s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.584642687 (139h28m47.28049144s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.591557684 (139h28m47.287406427s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.601517589 (139h28m47.297366333s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.688637153 (139h28m47.384485906s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:42.448516803 (139h29m14.144365566s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.448842534 (139h29m14.144691287s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: rpc: dial tcp 145.79.227.13:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449324783 (139h29m14.145173536s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449692342 (139h29m14.145541085s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450039493 (139h29m14.145888236s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450238873 (139h29m14.146087626s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450426109 (139h29m14.146274862s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450710671 (139h29m14.146559434s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: rpc: dial tcp 145.79.227.20:4437: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450894401 (139h29m14.146743154s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: rpc: dial tcp 145.79.227.25:4617: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.451068282 (139h29m14.146917036s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.453469054 (139h29m14.149317797s) [seg_pos={0 0}, upload=122910]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:43.162545170 (139h29m14.858393923s) [upload=122910]: Done uploading segments... Nov 5 07:09:43.162601467 (139h29m14.85845022s) [upload=122910]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 5.945450738s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.932807124s trace: https://jaeger.dev.storj.io/trace/75afb5f91d3935bc FAIL (2025-11-05 07:40:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== v 5 07:09:30.093538136 (139h29m1.789386889s) [upload=122911]: Uploading segments... Nov 5 07:09:30.093576248 (139h29m1.789424992s) [upload=122911]: Got next segment. Inline: false Nov 5 07:09:30.093604072 (139h29m1.789452825s) [upload=122911]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 07:09:30.093608600 (139h29m1.789457343s) [upload=122911]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:30.339289624 (139h29m2.035138367s) [seg_pos={0 0}, upload=122911]: Begin upload segment... Nov 5 07:09:30.339349428 (139h29m2.035198181s) [seg_pos={0 0}, upload=122911]: Done begin upload segment. Nov 5 07:09:30.339386408 (139h29m2.035235151s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340339152 (139h29m2.036187905s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340990803 (139h29m2.036839556s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341444226 (139h29m2.03729298s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341919592 (139h29m2.037768375s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342371553 (139h29m2.038220306s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342835706 (139h29m2.038684449s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343494230 (139h29m2.039342973s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343970537 (139h29m2.03981929s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.345438873 (139h29m2.041287626s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.347000497 (139h29m2.04284925s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.348576918 (139h29m2.044425661s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.350325538 (139h29m2.046174281s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.351367712 (139h29m2.047216465s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.354636787 (139h29m2.050485571s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.356330162 (139h29m2.052178915s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357738744 (139h29m2.053587487s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357956329 (139h29m2.053805172s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:31.727744519 (139h29m3.423593262s) [upload=122911]: Next returned nil segment Nov 5 07:09:31.727759807 (139h29m3.42360856s) [upload=122911]: Waiting for error group managing segments... Nov 5 07:09:32.079494147 (139h29m3.77534289s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.084663190 (139h29m3.780511933s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.140346892 (139h29m3.836195645s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.141263718 (139h29m3.837112472s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.325083005 (139h29m4.020931758s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.327962548 (139h29m4.023811301s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.375450033 (139h29m4.071298786s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.421783842 (139h29m4.117632595s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.435408857 (139h29m4.13125761s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.450128345 (139h29m4.145977088s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.464432925 (139h29m4.160281678s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.483329487 (139h29m4.17917824s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.594589918 (139h29m4.290438661s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:10:29.406200389 (139h30m1.102049142s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:145.79.227.13:4820): protocol: rpc: dial tcp 145.79.227.13:4820: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406293496 (139h30m1.102142249s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb, address:145.79.227.16:4920): protocol: rpc: dial tcp 145.79.227.16:4920: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406452028 (139h30m1.102300781s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:145.79.227.20:4440): protocol: rpc: dial tcp 145.79.227.20:4440: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406646058 (139h30m1.102494801s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:145.79.227.39:4047): protocol: rpc: dial tcp 145.79.227.39:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406963362 (139h30m1.102812125s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: rpc: dial tcp 145.79.227.2:4131: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.407007867 (139h30m1.10285662s) [seg_pos={0 0}, upload=122911]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 5 07:10:29.648039207 (139h30m1.34388797s) [upload=122911]: Done uploading segments... Nov 5 07:10:29.648083852 (139h30m1.343932595s) [upload=122911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.932837692s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................XX.....................................] duration: 5.688443883s trace: https://jaeger.dev.storj.io/trace/fe05d2f05e7af5f FAIL (2025-11-05 07:40:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 164056 (139h28m44.854012819s) [upload=122910]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:13.414175591 (139h28m45.110024344s) [seg_pos={0 0}, upload=122910]: Begin upload segment... Nov 5 07:09:13.414231787 (139h28m45.110080541s) [seg_pos={0 0}, upload=122910]: Done begin upload segment. Nov 5 07:09:13.414274930 (139h28m45.110123683s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415327825 (139h28m45.111176578s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415861491 (139h28m45.111710244s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416375510 (139h28m45.112224263s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416678197 (139h28m45.11252695s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417009026 (139h28m45.112857779s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417341269 (139h28m45.113190022s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417656319 (139h28m45.113505072s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.418114902 (139h28m45.113963645s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.420954530 (139h28m45.116803273s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.422854929 (139h28m45.118703682s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.424915342 (139h28m45.120764095s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.426819858 (139h28m45.122668611s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.428917523 (139h28m45.124766276s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.430615777 (139h28m45.12646452s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.432400986 (139h28m45.128249739s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:15.183539119 (139h28m46.879387872s) [upload=122910]: Next returned nil segment Nov 5 07:09:15.183553485 (139h28m46.879402239s) [upload=122910]: Waiting for error group managing segments... Nov 5 07:09:15.515060579 (139h28m47.210909332s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.562983674 (139h28m47.258832417s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.584642687 (139h28m47.28049144s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.591557684 (139h28m47.287406427s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.601517589 (139h28m47.297366333s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.688637153 (139h28m47.384485906s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:42.448516803 (139h29m14.144365566s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.448842534 (139h29m14.144691287s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: rpc: dial tcp 145.79.227.13:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449324783 (139h29m14.145173536s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449692342 (139h29m14.145541085s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450039493 (139h29m14.145888236s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450238873 (139h29m14.146087626s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450426109 (139h29m14.146274862s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450710671 (139h29m14.146559434s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: rpc: dial tcp 145.79.227.20:4437: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450894401 (139h29m14.146743154s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: rpc: dial tcp 145.79.227.25:4617: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.451068282 (139h29m14.146917036s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.453469054 (139h29m14.149317797s) [seg_pos={0 0}, upload=122910]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:43.162545170 (139h29m14.858393923s) [upload=122910]: Done uploading segments... Nov 5 07:09:43.162601467 (139h29m14.85845022s) [upload=122910]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 5.688491654s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.494618502s trace: https://jaeger.dev.storj.io/trace/1b48011ff2f3bc78 FAIL (2025-11-05 07:40:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.494652516s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.857919942s trace: https://jaeger.dev.storj.io/trace/75afb5f91d3935bc FAIL (2025-11-05 07:39:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== v 5 07:09:30.093538136 (139h29m1.789386889s) [upload=122911]: Uploading segments... Nov 5 07:09:30.093576248 (139h29m1.789424992s) [upload=122911]: Got next segment. Inline: false Nov 5 07:09:30.093604072 (139h29m1.789452825s) [upload=122911]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 07:09:30.093608600 (139h29m1.789457343s) [upload=122911]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:30.339289624 (139h29m2.035138367s) [seg_pos={0 0}, upload=122911]: Begin upload segment... Nov 5 07:09:30.339349428 (139h29m2.035198181s) [seg_pos={0 0}, upload=122911]: Done begin upload segment. Nov 5 07:09:30.339386408 (139h29m2.035235151s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340339152 (139h29m2.036187905s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340990803 (139h29m2.036839556s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341444226 (139h29m2.03729298s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341919592 (139h29m2.037768375s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342371553 (139h29m2.038220306s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342835706 (139h29m2.038684449s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343494230 (139h29m2.039342973s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343970537 (139h29m2.03981929s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.345438873 (139h29m2.041287626s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.347000497 (139h29m2.04284925s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.348576918 (139h29m2.044425661s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.350325538 (139h29m2.046174281s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.351367712 (139h29m2.047216465s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.354636787 (139h29m2.050485571s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.356330162 (139h29m2.052178915s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357738744 (139h29m2.053587487s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357956329 (139h29m2.053805172s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:31.727744519 (139h29m3.423593262s) [upload=122911]: Next returned nil segment Nov 5 07:09:31.727759807 (139h29m3.42360856s) [upload=122911]: Waiting for error group managing segments... Nov 5 07:09:32.079494147 (139h29m3.77534289s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.084663190 (139h29m3.780511933s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.140346892 (139h29m3.836195645s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.141263718 (139h29m3.837112472s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.325083005 (139h29m4.020931758s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.327962548 (139h29m4.023811301s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.375450033 (139h29m4.071298786s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.421783842 (139h29m4.117632595s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.435408857 (139h29m4.13125761s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.450128345 (139h29m4.145977088s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.464432925 (139h29m4.160281678s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.483329487 (139h29m4.17917824s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.594589918 (139h29m4.290438661s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:10:29.406200389 (139h30m1.102049142s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:145.79.227.13:4820): protocol: rpc: dial tcp 145.79.227.13:4820: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406293496 (139h30m1.102142249s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb, address:145.79.227.16:4920): protocol: rpc: dial tcp 145.79.227.16:4920: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406452028 (139h30m1.102300781s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:145.79.227.20:4440): protocol: rpc: dial tcp 145.79.227.20:4440: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406646058 (139h30m1.102494801s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:145.79.227.39:4047): protocol: rpc: dial tcp 145.79.227.39:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406963362 (139h30m1.102812125s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: rpc: dial tcp 145.79.227.2:4131: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.407007867 (139h30m1.10285662s) [seg_pos={0 0}, upload=122911]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 5 07:10:29.648039207 (139h30m1.34388797s) [upload=122911]: Done uploading segments... Nov 5 07:10:29.648083852 (139h30m1.343932595s) [upload=122911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.857947776s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................XX....................................] duration: 5.724794145s trace: https://jaeger.dev.storj.io/trace/fe05d2f05e7af5f FAIL (2025-11-05 07:39:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 164056 (139h28m44.854012819s) [upload=122910]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:13.414175591 (139h28m45.110024344s) [seg_pos={0 0}, upload=122910]: Begin upload segment... Nov 5 07:09:13.414231787 (139h28m45.110080541s) [seg_pos={0 0}, upload=122910]: Done begin upload segment. Nov 5 07:09:13.414274930 (139h28m45.110123683s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415327825 (139h28m45.111176578s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415861491 (139h28m45.111710244s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416375510 (139h28m45.112224263s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416678197 (139h28m45.11252695s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417009026 (139h28m45.112857779s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417341269 (139h28m45.113190022s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417656319 (139h28m45.113505072s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.418114902 (139h28m45.113963645s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.420954530 (139h28m45.116803273s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.422854929 (139h28m45.118703682s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.424915342 (139h28m45.120764095s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.426819858 (139h28m45.122668611s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.428917523 (139h28m45.124766276s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.430615777 (139h28m45.12646452s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.432400986 (139h28m45.128249739s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:15.183539119 (139h28m46.879387872s) [upload=122910]: Next returned nil segment Nov 5 07:09:15.183553485 (139h28m46.879402239s) [upload=122910]: Waiting for error group managing segments... Nov 5 07:09:15.515060579 (139h28m47.210909332s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.562983674 (139h28m47.258832417s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.584642687 (139h28m47.28049144s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.591557684 (139h28m47.287406427s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.601517589 (139h28m47.297366333s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.688637153 (139h28m47.384485906s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:42.448516803 (139h29m14.144365566s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.448842534 (139h29m14.144691287s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: rpc: dial tcp 145.79.227.13:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449324783 (139h29m14.145173536s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449692342 (139h29m14.145541085s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450039493 (139h29m14.145888236s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450238873 (139h29m14.146087626s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450426109 (139h29m14.146274862s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450710671 (139h29m14.146559434s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: rpc: dial tcp 145.79.227.20:4437: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450894401 (139h29m14.146743154s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: rpc: dial tcp 145.79.227.25:4617: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.451068282 (139h29m14.146917036s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.453469054 (139h29m14.149317797s) [seg_pos={0 0}, upload=122910]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:43.162545170 (139h29m14.858393923s) [upload=122910]: Done uploading segments... Nov 5 07:09:43.162601467 (139h29m14.85845022s) [upload=122910]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 5.724824983s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.164988632s trace: https://jaeger.dev.storj.io/trace/1b48011ff2f3bc78 FAIL (2025-11-05 07:39:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.165009941s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.166420691s trace: https://jaeger.dev.storj.io/trace/1b48011ff2f3bc78 FAIL (2025-11-05 07:38:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.166449776s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................XX...................................] duration: 5.810358509s trace: https://jaeger.dev.storj.io/trace/fe05d2f05e7af5f FAIL (2025-11-05 07:38:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 164056 (139h28m44.854012819s) [upload=122910]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:13.414175591 (139h28m45.110024344s) [seg_pos={0 0}, upload=122910]: Begin upload segment... Nov 5 07:09:13.414231787 (139h28m45.110080541s) [seg_pos={0 0}, upload=122910]: Done begin upload segment. Nov 5 07:09:13.414274930 (139h28m45.110123683s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415327825 (139h28m45.111176578s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.415861491 (139h28m45.111710244s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416375510 (139h28m45.112224263s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.416678197 (139h28m45.11252695s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417009026 (139h28m45.112857779s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417341269 (139h28m45.113190022s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.417656319 (139h28m45.113505072s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.418114902 (139h28m45.113963645s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.420954530 (139h28m45.116803273s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.422854929 (139h28m45.118703682s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.424915342 (139h28m45.120764095s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.426819858 (139h28m45.122668611s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.428917523 (139h28m45.124766276s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.430615777 (139h28m45.12646452s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:13.432400986 (139h28m45.128249739s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Uploading piece... Nov 5 07:09:15.183539119 (139h28m46.879387872s) [upload=122910]: Next returned nil segment Nov 5 07:09:15.183553485 (139h28m46.879402239s) [upload=122910]: Waiting for error group managing segments... Nov 5 07:09:15.515060579 (139h28m47.210909332s) [noise=true , address=45.91.116.6:4975 , piece_id=SOGYACQKDKVOQ2CFPO72LP4WVJZO4FQWHEBDE3AQVYX46V6UWQVQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.562983674 (139h28m47.258832417s) [noise=true , address=77.90.31.14:4663 , piece_id=YHV4SKT5RJLQRXUIOVJ23VXGWY3EKHDEQLIALH4KZE6PJP2MNBRA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.584642687 (139h28m47.28049144s) [noise=true , address=45.91.116.4:4937 , piece_id=E6ZLRZDKSEMNFQTNTGLOZWWNVJMFO46YMYJKWZ2DVT347YPRHKGQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.591557684 (139h28m47.287406427s) [noise=true , address=77.90.31.13:4628 , piece_id=5S73X2KZO7E4OYMCPP6FOXGQPN2Y32IQ2ZOMFHBXGANJC45JNDQQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.601517589 (139h28m47.297366333s) [noise=true , address=77.90.31.10:4517 , piece_id=ADPKVLGR6ECRYL6TQWFCOPGPYQSYN4BD6FUQNEONDRI337WYJMEA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:15.688637153 (139h28m47.384485906s) [noise=true , address=77.90.31.21:4879 , piece_id=5P4PB3WVTRJWEKGG4WNDVGPMH3VZJHA6YLBV3JME72PZ7G27465A, seg_pos={0 0}, upload=122910]: Done uploading piece. err: <nil> Nov 5 07:09:42.448516803 (139h29m14.144365566s) [noise=true , address=145.79.227.14:4837 , piece_id=76F7OJH2MK4RYNWEF3DO4EFHOY2UBBMOS64OW4PS5VOJXPQE6QZQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.448842534 (139h29m14.144691287s) [noise=true , address=145.79.227.13:4803 , piece_id=APINC3457FNIYG7JZGUX5FF4GAVJGAWQPTPYRJF2WCKOZADAGPHA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: rpc: dial tcp 145.79.227.13:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449324783 (139h29m14.145173536s) [noise=true , address=145.79.227.21:4475 , piece_id=GOUDLNB2WWV4LUPGL5GSOOEA53ZQAFDIQANWP6WDA3FJ3SPBHFNA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.449692342 (139h29m14.145541085s) [noise=true , address=145.79.227.8:4329 , piece_id=ORU7HSG77K7HBPCI5HYG7FS5E6UPW3PTVRVTUE7YOQ5WWF3BE2QA, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450039493 (139h29m14.145888236s) [noise=true , address=145.79.227.8:4331 , piece_id=SSI572JR3MQHGBT67BVGSOG2XTS7JU242APHXX3NLLWHGADV6M6Q, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: rpc: dial tcp 145.79.227.8:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450238873 (139h29m14.146087626s) [noise=true , address=145.79.227.16:4909 , piece_id=VJWUOPUF7L6UU4YHHTJ2PL36ZFXFRNIXIUSPPO7X2XD5GUTYSMCQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450426109 (139h29m14.146274862s) [noise=true , address=145.79.227.18:4005 , piece_id=7QMCUG3ICKGBUF3KVZOGKZGZAWDAN25Y4XGUCTRKOWHTDTRSEWNQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450710671 (139h29m14.146559434s) [noise=true , address=145.79.227.20:4437 , piece_id=ZSOJWEPDCM53UT3NWOHRYANYBVVX3XCSLYDSL2WY4Q2UQQMFHLIQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: rpc: dial tcp 145.79.227.20:4437: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.450894401 (139h29m14.146743154s) [noise=true , address=145.79.227.25:4617 , piece_id=V6PGWJIJBTU7GLL5K2SK7SDWP7QY6C4CENKU7ZZPAWXMT3U4HNFQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: rpc: dial tcp 145.79.227.25:4617: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.451068282 (139h29m14.146917036s) [noise=true , address=145.79.227.11:4729 , piece_id=J7ADO4E5DOHMIOOSLJDYVLLYQBB4HQLXFCJKIERKZPQCPJLB2YMQ, seg_pos={0 0}, upload=122910]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:42.453469054 (139h29m14.149317797s) [seg_pos={0 0}, upload=122910]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:43.162545170 (139h29m14.858393923s) [upload=122910]: Done uploading segments... Nov 5 07:09:43.162601467 (139h29m14.85845022s) [upload=122910]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 5.810396111s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.174604193s trace: https://jaeger.dev.storj.io/trace/75afb5f91d3935bc FAIL (2025-11-05 07:38:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== v 5 07:09:30.093538136 (139h29m1.789386889s) [upload=122911]: Uploading segments... Nov 5 07:09:30.093576248 (139h29m1.789424992s) [upload=122911]: Got next segment. Inline: false Nov 5 07:09:30.093604072 (139h29m1.789452825s) [upload=122911]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 07:09:30.093608600 (139h29m1.789457343s) [upload=122911]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:09:30.339289624 (139h29m2.035138367s) [seg_pos={0 0}, upload=122911]: Begin upload segment... Nov 5 07:09:30.339349428 (139h29m2.035198181s) [seg_pos={0 0}, upload=122911]: Done begin upload segment. Nov 5 07:09:30.339386408 (139h29m2.035235151s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340339152 (139h29m2.036187905s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.340990803 (139h29m2.036839556s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341444226 (139h29m2.03729298s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.341919592 (139h29m2.037768375s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342371553 (139h29m2.038220306s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.342835706 (139h29m2.038684449s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343494230 (139h29m2.039342973s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.343970537 (139h29m2.03981929s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.345438873 (139h29m2.041287626s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.347000497 (139h29m2.04284925s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.348576918 (139h29m2.044425661s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.350325538 (139h29m2.046174281s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.351367712 (139h29m2.047216465s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.354636787 (139h29m2.050485571s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.356330162 (139h29m2.052178915s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357738744 (139h29m2.053587487s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:30.357956329 (139h29m2.053805172s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Uploading piece... Nov 5 07:09:31.727744519 (139h29m3.423593262s) [upload=122911]: Next returned nil segment Nov 5 07:09:31.727759807 (139h29m3.42360856s) [upload=122911]: Waiting for error group managing segments... Nov 5 07:09:32.079494147 (139h29m3.77534289s) [noise=true , address=5.231.73.2:4004 , piece_id=AZDOOBEDZG3EQQEQBMM76JZLG7VXM3MLHTO4MNA6L2MELW3UQLYA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.084663190 (139h29m3.780511933s) [noise=true , address=5.231.73.3:4015 , piece_id=IIL6QBHEDJISXFUL7NEVFCDJT7GEOJPNMDO7I35WV4HQCQXXMZCQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.140346892 (139h29m3.836195645s) [noise=true , address=5.231.73.4:4034 , piece_id=DUNHFMSAL5TMZX23NCGHRUBQ2RO5IGOBZ57IJRDEHS6VN7GK4GYQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.141263718 (139h29m3.837112472s) [noise=true , address=5.231.73.5:4012 , piece_id=YBMPMBHXGNY4LWCOK22P3EWAGOHYNBNA4ERRDVOM2WKAIYF33YJQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.325083005 (139h29m4.020931758s) [noise=true , address=45.91.116.5:4322 , piece_id=Y2FNO67QNV7CTT3NMZH4HWH2M5TA4VS5EWQZO2BUX4JLQ3CAL2CA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.327962548 (139h29m4.023811301s) [noise=true , address=45.91.116.4:4963 , piece_id=MKBMA3NJEJ6OWDXRSDY63USBZVTHXDTW3QZ64KSPC3RVOEIGFQTA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.375450033 (139h29m4.071298786s) [noise=true , address=77.90.31.17:4785 , piece_id=N5MCSRO73YLZA53JRNMC2PMKJKBYGK6C247BAPRF42VGCW5S3E3A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.421783842 (139h29m4.117632595s) [noise=true , address=77.90.31.11:4559 , piece_id=4LNUAPP6QSX54A4L2ZPSYWPKRLZGBKNZQWB3N6JR3ZRKNSIDX7SQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.435408857 (139h29m4.13125761s) [noise=true , address=77.90.31.10:4547 , piece_id=AAOURACLDOD3LYH5KQRRXM3XMGWVNZG4GCODBVPUSRQYDDZWELEA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.450128345 (139h29m4.145977088s) [noise=true , address=77.90.31.18:4816 , piece_id=WQ5GKIMP66WCWLUT2B6SMJQUC3QJXXJQG5Y7JZUQO7LNPXDXTTNQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.464432925 (139h29m4.160281678s) [noise=true , address=45.91.116.13:4176 , piece_id=HVQJMLB7AO2JSGZBLXL5XB2IIEWMEZIFRXWKR6YVMYMSITMSUDLA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.483329487 (139h29m4.17917824s) [noise=true , address=77.90.31.14:4682 , piece_id=4TDAF4AWOEPVUXEHJIYTGCILCWH4NVHFS3C2UUIIS6MLPKNXNHIQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:09:32.594589918 (139h29m4.290438661s) [noise=true , address=77.90.31.15:4707 , piece_id=7AAJYAIOJZ5ASML7IYRTZFX4PURUSYO4JXKUIF3YLLVE3YIFMEOQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: <nil> Nov 5 07:10:29.406200389 (139h30m1.102049142s) [noise=true , address=145.79.227.13:4820 , piece_id=6PBLZK6TCIIIZZNUTXB67VJB7BT7YMVYTSXDTRMOL4KIOZSFHUBQ, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:145.79.227.13:4820): protocol: rpc: dial tcp 145.79.227.13:4820: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406293496 (139h30m1.102142249s) [noise=true , address=145.79.227.16:4920 , piece_id=5CW3A7C3QAW5G2KSPWSON2V3RVRBD4ESFKA2Q7LEC5J6GK4XYCOA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb, address:145.79.227.16:4920): protocol: rpc: dial tcp 145.79.227.16:4920: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406452028 (139h30m1.102300781s) [noise=true , address=145.79.227.20:4440 , piece_id=ZKGAC6UPEJZND37PC4ZOXQXGWWWZ3EPKEVS3PBY3HONGCSTDLI5A, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:145.79.227.20:4440): protocol: rpc: dial tcp 145.79.227.20:4440: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406646058 (139h30m1.102494801s) [noise=true , address=145.79.227.39:4047 , piece_id=LJHLRIVGRNSTXK7BRDUAT5SNH4Q644YWSIJKJIAYQ5HT2RDKC7SA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:145.79.227.39:4047): protocol: rpc: dial tcp 145.79.227.39:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.406963362 (139h30m1.102812125s) [noise=true , address=145.79.227.2:4131 , piece_id=GMWOCM6BBUITD2VRAZP7PAYHQKDGW4RL2WYW6N2SW3LLD4KUZ2OA, seg_pos={0 0}, upload=122911]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: rpc: dial tcp 145.79.227.2:4131: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:10:29.407007867 (139h30m1.10285662s) [seg_pos={0 0}, upload=122911]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 5 07:10:29.648039207 (139h30m1.34388797s) [upload=122911]: Done uploading segments... Nov 5 07:10:29.648083852 (139h30m1.343932595s) [upload=122911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.174641223s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.461528731s trace: https://jaeger.dev.storj.io/trace/1b48011ff2f3bc78 FAIL (2025-11-05 07:37:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.461548269s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)