Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............X......X................................XX.....] duration: 7.649177708s trace: https://jaeger.dev.storj.io/trace/32f48b4f1799253 FAIL (2025-11-11 17:13:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ) context deadline exceeded; piecestore: (Node ID: 12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, Piece ID: BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ) context deadline exceeded; piecestore: (Node ID: 12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, Piece ID: XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q) context deadline exceeded; piecestore: (Node ID: 12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, Piece ID: BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA) context deadline exceeded ======LOGS====== 11 17:09:17.808254746 (57h5m16.359725207s) [upload=50295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:09:17.808259335 (57h5m16.359729786s) [upload=50295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:09:18.065943994 (57h5m16.617414455s) [seg_pos={0 0}, upload=50295]: Begin upload segment... Nov 11 17:09:18.066007305 (57h5m16.617477766s) [seg_pos={0 0}, upload=50295]: Done begin upload segment. Nov 11 17:09:18.066048393 (57h5m16.617518854s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.067020474 (57h5m16.618490925s) [noise=true , address=145.79.227.14:4837 , piece_id=XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.067697985 (57h5m16.619168436s) [noise=true , address=45.91.116.11:4118 , piece_id=GLBZIWAUC6OBA3EXQSFDVPQZ3B7AOMA6PSBUJU7GU6CLHRW45NMA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.069137075 (57h5m16.620607536s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.070196713 (57h5m16.621667164s) [noise=true , address=145.79.227.6:4258 , piece_id=BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.074296229 (57h5m16.62576668s) [noise=true , address=45.91.116.4:4940 , piece_id=7HY6FK6RA4AERNEGFIIUNGRKG5UI6QJ6D6NATTFZKKWDSKTGY3PQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.076492341 (57h5m16.627962802s) [noise=true , address=145.79.227.17:4970 , piece_id=EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.078065547 (57h5m16.629536008s) [noise=true , address=145.79.227.9:4365 , piece_id=BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.080540098 (57h5m16.632010549s) [noise=true , address=45.91.116.5:4297 , piece_id=PF7UO5J6C5KHI34EIL4ITP37MSVVNGWW3YC63TJQPWSDNCGIEXYA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.081056011 (57h5m16.632526472s) [noise=true , address=45.91.116.6:4973 , piece_id=DGNIVMHVC476IJ4HVBKEGQQY2ADVWBF5TQ272H3EUBXDXMFRR7OQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.084048509 (57h5m16.63551897s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.085709442 (57h5m16.637179893s) [noise=true , address=77.90.31.12:4592 , piece_id=VRSDWRPJKAWWU3X2DYQSQB5QLTYPHFH6RMIRNSRKQIK75ZF2UAHA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.086524133 (57h5m16.637994594s) [noise=true , address=145.79.227.22:4509 , piece_id=DADDKGVC2EMAEIFHT5GQSQA2BZYWLSW2HFUFDCJWLKTPV5IRPDYQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.087118326 (57h5m16.638588777s) [noise=true , address=45.91.116.13:4153 , piece_id=NK22OZKAZRF746YPBHO27E6QGR2VJI33DAF5TW7FSUMDK7VIU5UQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.088483044 (57h5m16.639953505s) [noise=true , address=77.90.31.15:4697 , piece_id=BCMQWJVGOJ7CDB3WCCTICLSUI7YCO5EONPOB6AHD47M3NSDTW27Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.089429967 (57h5m16.640900428s) [noise=true , address=45.91.116.7:4009 , piece_id=B2ONDGKYMCRX3DSMNSKX34BRJLRXRJD35BKL4FE7FMUFKLGRGERQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:19.843494253 (57h5m18.394964714s) [upload=50295]: Next returned nil segment Nov 11 17:09:19.843513309 (57h5m18.39498379s) [upload=50295]: Waiting for error group managing segments... Nov 11 17:09:20.178022832 (57h5m18.729493333s) [noise=true , address=45.91.116.4:4940 , piece_id=7HY6FK6RA4AERNEGFIIUNGRKG5UI6QJ6D6NATTFZKKWDSKTGY3PQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.191212378 (57h5m18.742682849s) [noise=true , address=77.90.31.12:4592 , piece_id=VRSDWRPJKAWWU3X2DYQSQB5QLTYPHFH6RMIRNSRKQIK75ZF2UAHA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.214652943 (57h5m18.766123394s) [noise=true , address=45.91.116.11:4118 , piece_id=GLBZIWAUC6OBA3EXQSFDVPQZ3B7AOMA6PSBUJU7GU6CLHRW45NMA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.228934699 (57h5m18.78040515s) [noise=true , address=45.91.116.7:4009 , piece_id=B2ONDGKYMCRX3DSMNSKX34BRJLRXRJD35BKL4FE7FMUFKLGRGERQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.257611417 (57h5m18.809081878s) [noise=true , address=45.91.116.6:4973 , piece_id=DGNIVMHVC476IJ4HVBKEGQQY2ADVWBF5TQ272H3EUBXDXMFRR7OQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.286180058 (57h5m18.837650519s) [noise=true , address=45.91.116.13:4153 , piece_id=NK22OZKAZRF746YPBHO27E6QGR2VJI33DAF5TW7FSUMDK7VIU5UQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.361426805 (57h5m18.912897266s) [noise=true , address=45.91.116.5:4297 , piece_id=PF7UO5J6C5KHI34EIL4ITP37MSVVNGWW3YC63TJQPWSDNCGIEXYA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.375330360 (57h5m18.926800821s) [noise=true , address=77.90.31.15:4697 , piece_id=BCMQWJVGOJ7CDB3WCCTICLSUI7YCO5EONPOB6AHD47M3NSDTW27Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:31.518915065 (57h5m30.070385516s) [noise=true , address=145.79.227.22:4509 , piece_id=DADDKGVC2EMAEIFHT5GQSQA2BZYWLSW2HFUFDCJWLKTPV5IRPDYQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:34.758887248 (57h5m33.310357699s) [noise=true , address=145.79.227.9:4365 , piece_id=BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:35.165420831 (57h5m33.716891282s) [noise=true , address=145.79.227.6:4258 , piece_id=BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:35.922385673 (57h5m34.473856124s) [noise=true , address=145.79.227.17:4970 , piece_id=EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:44.840964696 (57h5m43.392435147s) [noise=true , address=145.79.227.14:4837 , piece_id=XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:44.841004482 (57h5m43.392474943s) [seg_pos={0 0}, upload=50295]: Segment reached optimal threshold of 13 pieces. Nov 11 17:09:44.841717200 (57h5m43.393187671s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection Nov 11 17:09:44.841729202 (57h5m43.393199683s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842051686 (57h5m43.393522147s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection Nov 11 17:09:44.842059662 (57h5m43.393530123s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842244383 (57h5m43.393714844s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection Nov 11 17:09:44.842252308 (57h5m43.393722759s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842269892 (57h5m43.393740353s) [seg_pos={0 0}, upload=50295]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 17:09:44.842338914 (57h5m43.393809375s) [upload=50295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 17:09:44.842343502 (57h5m43.393813963s) [upload=50295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 17:09:45.141069414 (57h5m43.692539875s) [upload=50295]: Done uploading segments... Nov 11 17:09:45.141088591 (57h5m43.692559042s) [upload=50295]: Upload finished. err: <nil> Duration: 7.64920943s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............X......X................................XX....] duration: 9.28668489s trace: https://jaeger.dev.storj.io/trace/32f48b4f1799253 FAIL (2025-11-11 17:13:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ) context deadline exceeded; piecestore: (Node ID: 12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, Piece ID: BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ) context deadline exceeded; piecestore: (Node ID: 12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, Piece ID: XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q) context deadline exceeded; piecestore: (Node ID: 12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, Piece ID: BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA) context deadline exceeded ======LOGS====== 11 17:09:17.808254746 (57h5m16.359725207s) [upload=50295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:09:17.808259335 (57h5m16.359729786s) [upload=50295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:09:18.065943994 (57h5m16.617414455s) [seg_pos={0 0}, upload=50295]: Begin upload segment... Nov 11 17:09:18.066007305 (57h5m16.617477766s) [seg_pos={0 0}, upload=50295]: Done begin upload segment. Nov 11 17:09:18.066048393 (57h5m16.617518854s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.067020474 (57h5m16.618490925s) [noise=true , address=145.79.227.14:4837 , piece_id=XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.067697985 (57h5m16.619168436s) [noise=true , address=45.91.116.11:4118 , piece_id=GLBZIWAUC6OBA3EXQSFDVPQZ3B7AOMA6PSBUJU7GU6CLHRW45NMA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.069137075 (57h5m16.620607536s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.070196713 (57h5m16.621667164s) [noise=true , address=145.79.227.6:4258 , piece_id=BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.074296229 (57h5m16.62576668s) [noise=true , address=45.91.116.4:4940 , piece_id=7HY6FK6RA4AERNEGFIIUNGRKG5UI6QJ6D6NATTFZKKWDSKTGY3PQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.076492341 (57h5m16.627962802s) [noise=true , address=145.79.227.17:4970 , piece_id=EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.078065547 (57h5m16.629536008s) [noise=true , address=145.79.227.9:4365 , piece_id=BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.080540098 (57h5m16.632010549s) [noise=true , address=45.91.116.5:4297 , piece_id=PF7UO5J6C5KHI34EIL4ITP37MSVVNGWW3YC63TJQPWSDNCGIEXYA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.081056011 (57h5m16.632526472s) [noise=true , address=45.91.116.6:4973 , piece_id=DGNIVMHVC476IJ4HVBKEGQQY2ADVWBF5TQ272H3EUBXDXMFRR7OQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.084048509 (57h5m16.63551897s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.085709442 (57h5m16.637179893s) [noise=true , address=77.90.31.12:4592 , piece_id=VRSDWRPJKAWWU3X2DYQSQB5QLTYPHFH6RMIRNSRKQIK75ZF2UAHA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.086524133 (57h5m16.637994594s) [noise=true , address=145.79.227.22:4509 , piece_id=DADDKGVC2EMAEIFHT5GQSQA2BZYWLSW2HFUFDCJWLKTPV5IRPDYQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.087118326 (57h5m16.638588777s) [noise=true , address=45.91.116.13:4153 , piece_id=NK22OZKAZRF746YPBHO27E6QGR2VJI33DAF5TW7FSUMDK7VIU5UQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.088483044 (57h5m16.639953505s) [noise=true , address=77.90.31.15:4697 , piece_id=BCMQWJVGOJ7CDB3WCCTICLSUI7YCO5EONPOB6AHD47M3NSDTW27Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.089429967 (57h5m16.640900428s) [noise=true , address=45.91.116.7:4009 , piece_id=B2ONDGKYMCRX3DSMNSKX34BRJLRXRJD35BKL4FE7FMUFKLGRGERQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:19.843494253 (57h5m18.394964714s) [upload=50295]: Next returned nil segment Nov 11 17:09:19.843513309 (57h5m18.39498379s) [upload=50295]: Waiting for error group managing segments... Nov 11 17:09:20.178022832 (57h5m18.729493333s) [noise=true , address=45.91.116.4:4940 , piece_id=7HY6FK6RA4AERNEGFIIUNGRKG5UI6QJ6D6NATTFZKKWDSKTGY3PQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.191212378 (57h5m18.742682849s) [noise=true , address=77.90.31.12:4592 , piece_id=VRSDWRPJKAWWU3X2DYQSQB5QLTYPHFH6RMIRNSRKQIK75ZF2UAHA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.214652943 (57h5m18.766123394s) [noise=true , address=45.91.116.11:4118 , piece_id=GLBZIWAUC6OBA3EXQSFDVPQZ3B7AOMA6PSBUJU7GU6CLHRW45NMA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.228934699 (57h5m18.78040515s) [noise=true , address=45.91.116.7:4009 , piece_id=B2ONDGKYMCRX3DSMNSKX34BRJLRXRJD35BKL4FE7FMUFKLGRGERQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.257611417 (57h5m18.809081878s) [noise=true , address=45.91.116.6:4973 , piece_id=DGNIVMHVC476IJ4HVBKEGQQY2ADVWBF5TQ272H3EUBXDXMFRR7OQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.286180058 (57h5m18.837650519s) [noise=true , address=45.91.116.13:4153 , piece_id=NK22OZKAZRF746YPBHO27E6QGR2VJI33DAF5TW7FSUMDK7VIU5UQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.361426805 (57h5m18.912897266s) [noise=true , address=45.91.116.5:4297 , piece_id=PF7UO5J6C5KHI34EIL4ITP37MSVVNGWW3YC63TJQPWSDNCGIEXYA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.375330360 (57h5m18.926800821s) [noise=true , address=77.90.31.15:4697 , piece_id=BCMQWJVGOJ7CDB3WCCTICLSUI7YCO5EONPOB6AHD47M3NSDTW27Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:31.518915065 (57h5m30.070385516s) [noise=true , address=145.79.227.22:4509 , piece_id=DADDKGVC2EMAEIFHT5GQSQA2BZYWLSW2HFUFDCJWLKTPV5IRPDYQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:34.758887248 (57h5m33.310357699s) [noise=true , address=145.79.227.9:4365 , piece_id=BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:35.165420831 (57h5m33.716891282s) [noise=true , address=145.79.227.6:4258 , piece_id=BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:35.922385673 (57h5m34.473856124s) [noise=true , address=145.79.227.17:4970 , piece_id=EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:44.840964696 (57h5m43.392435147s) [noise=true , address=145.79.227.14:4837 , piece_id=XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:44.841004482 (57h5m43.392474943s) [seg_pos={0 0}, upload=50295]: Segment reached optimal threshold of 13 pieces. Nov 11 17:09:44.841717200 (57h5m43.393187671s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection Nov 11 17:09:44.841729202 (57h5m43.393199683s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842051686 (57h5m43.393522147s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection Nov 11 17:09:44.842059662 (57h5m43.393530123s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842244383 (57h5m43.393714844s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection Nov 11 17:09:44.842252308 (57h5m43.393722759s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842269892 (57h5m43.393740353s) [seg_pos={0 0}, upload=50295]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 17:09:44.842338914 (57h5m43.393809375s) [upload=50295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 17:09:44.842343502 (57h5m43.393813963s) [upload=50295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 17:09:45.141069414 (57h5m43.692539875s) [upload=50295]: Done uploading segments... Nov 11 17:09:45.141088591 (57h5m43.692559042s) [upload=50295]: Upload finished. err: <nil> Duration: 9.286709696s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 8.578137154s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 17:12:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 8.578167321s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............X......X................................XX...] duration: 7.518220738s trace: https://jaeger.dev.storj.io/trace/32f48b4f1799253 FAIL (2025-11-11 17:12:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ) context deadline exceeded; piecestore: (Node ID: 12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, Piece ID: BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ) context deadline exceeded; piecestore: (Node ID: 12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, Piece ID: XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q) context deadline exceeded; piecestore: (Node ID: 12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, Piece ID: BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA) context deadline exceeded ======LOGS====== 11 17:09:17.808254746 (57h5m16.359725207s) [upload=50295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:09:17.808259335 (57h5m16.359729786s) [upload=50295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:09:18.065943994 (57h5m16.617414455s) [seg_pos={0 0}, upload=50295]: Begin upload segment... Nov 11 17:09:18.066007305 (57h5m16.617477766s) [seg_pos={0 0}, upload=50295]: Done begin upload segment. Nov 11 17:09:18.066048393 (57h5m16.617518854s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.067020474 (57h5m16.618490925s) [noise=true , address=145.79.227.14:4837 , piece_id=XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.067697985 (57h5m16.619168436s) [noise=true , address=45.91.116.11:4118 , piece_id=GLBZIWAUC6OBA3EXQSFDVPQZ3B7AOMA6PSBUJU7GU6CLHRW45NMA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.069137075 (57h5m16.620607536s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.070196713 (57h5m16.621667164s) [noise=true , address=145.79.227.6:4258 , piece_id=BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.074296229 (57h5m16.62576668s) [noise=true , address=45.91.116.4:4940 , piece_id=7HY6FK6RA4AERNEGFIIUNGRKG5UI6QJ6D6NATTFZKKWDSKTGY3PQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.076492341 (57h5m16.627962802s) [noise=true , address=145.79.227.17:4970 , piece_id=EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.078065547 (57h5m16.629536008s) [noise=true , address=145.79.227.9:4365 , piece_id=BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.080540098 (57h5m16.632010549s) [noise=true , address=45.91.116.5:4297 , piece_id=PF7UO5J6C5KHI34EIL4ITP37MSVVNGWW3YC63TJQPWSDNCGIEXYA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.081056011 (57h5m16.632526472s) [noise=true , address=45.91.116.6:4973 , piece_id=DGNIVMHVC476IJ4HVBKEGQQY2ADVWBF5TQ272H3EUBXDXMFRR7OQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.084048509 (57h5m16.63551897s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.085709442 (57h5m16.637179893s) [noise=true , address=77.90.31.12:4592 , piece_id=VRSDWRPJKAWWU3X2DYQSQB5QLTYPHFH6RMIRNSRKQIK75ZF2UAHA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.086524133 (57h5m16.637994594s) [noise=true , address=145.79.227.22:4509 , piece_id=DADDKGVC2EMAEIFHT5GQSQA2BZYWLSW2HFUFDCJWLKTPV5IRPDYQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.087118326 (57h5m16.638588777s) [noise=true , address=45.91.116.13:4153 , piece_id=NK22OZKAZRF746YPBHO27E6QGR2VJI33DAF5TW7FSUMDK7VIU5UQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.088483044 (57h5m16.639953505s) [noise=true , address=77.90.31.15:4697 , piece_id=BCMQWJVGOJ7CDB3WCCTICLSUI7YCO5EONPOB6AHD47M3NSDTW27Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.089429967 (57h5m16.640900428s) [noise=true , address=45.91.116.7:4009 , piece_id=B2ONDGKYMCRX3DSMNSKX34BRJLRXRJD35BKL4FE7FMUFKLGRGERQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:19.843494253 (57h5m18.394964714s) [upload=50295]: Next returned nil segment Nov 11 17:09:19.843513309 (57h5m18.39498379s) [upload=50295]: Waiting for error group managing segments... Nov 11 17:09:20.178022832 (57h5m18.729493333s) [noise=true , address=45.91.116.4:4940 , piece_id=7HY6FK6RA4AERNEGFIIUNGRKG5UI6QJ6D6NATTFZKKWDSKTGY3PQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.191212378 (57h5m18.742682849s) [noise=true , address=77.90.31.12:4592 , piece_id=VRSDWRPJKAWWU3X2DYQSQB5QLTYPHFH6RMIRNSRKQIK75ZF2UAHA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.214652943 (57h5m18.766123394s) [noise=true , address=45.91.116.11:4118 , piece_id=GLBZIWAUC6OBA3EXQSFDVPQZ3B7AOMA6PSBUJU7GU6CLHRW45NMA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.228934699 (57h5m18.78040515s) [noise=true , address=45.91.116.7:4009 , piece_id=B2ONDGKYMCRX3DSMNSKX34BRJLRXRJD35BKL4FE7FMUFKLGRGERQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.257611417 (57h5m18.809081878s) [noise=true , address=45.91.116.6:4973 , piece_id=DGNIVMHVC476IJ4HVBKEGQQY2ADVWBF5TQ272H3EUBXDXMFRR7OQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.286180058 (57h5m18.837650519s) [noise=true , address=45.91.116.13:4153 , piece_id=NK22OZKAZRF746YPBHO27E6QGR2VJI33DAF5TW7FSUMDK7VIU5UQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.361426805 (57h5m18.912897266s) [noise=true , address=45.91.116.5:4297 , piece_id=PF7UO5J6C5KHI34EIL4ITP37MSVVNGWW3YC63TJQPWSDNCGIEXYA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.375330360 (57h5m18.926800821s) [noise=true , address=77.90.31.15:4697 , piece_id=BCMQWJVGOJ7CDB3WCCTICLSUI7YCO5EONPOB6AHD47M3NSDTW27Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:31.518915065 (57h5m30.070385516s) [noise=true , address=145.79.227.22:4509 , piece_id=DADDKGVC2EMAEIFHT5GQSQA2BZYWLSW2HFUFDCJWLKTPV5IRPDYQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:34.758887248 (57h5m33.310357699s) [noise=true , address=145.79.227.9:4365 , piece_id=BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:35.165420831 (57h5m33.716891282s) [noise=true , address=145.79.227.6:4258 , piece_id=BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:35.922385673 (57h5m34.473856124s) [noise=true , address=145.79.227.17:4970 , piece_id=EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:44.840964696 (57h5m43.392435147s) [noise=true , address=145.79.227.14:4837 , piece_id=XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:44.841004482 (57h5m43.392474943s) [seg_pos={0 0}, upload=50295]: Segment reached optimal threshold of 13 pieces. Nov 11 17:09:44.841717200 (57h5m43.393187671s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection Nov 11 17:09:44.841729202 (57h5m43.393199683s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842051686 (57h5m43.393522147s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection Nov 11 17:09:44.842059662 (57h5m43.393530123s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842244383 (57h5m43.393714844s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection Nov 11 17:09:44.842252308 (57h5m43.393722759s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842269892 (57h5m43.393740353s) [seg_pos={0 0}, upload=50295]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 17:09:44.842338914 (57h5m43.393809375s) [upload=50295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 17:09:44.842343502 (57h5m43.393813963s) [upload=50295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 17:09:45.141069414 (57h5m43.692539875s) [upload=50295]: Done uploading segments... Nov 11 17:09:45.141088591 (57h5m43.692559042s) [upload=50295]: Upload finished. err: <nil> Duration: 7.518865145s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 7.64884108s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 17:12:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 7.648876106s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.236631551s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 17:11:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.236724328s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................X......X................................XX..] duration: 8.324957899s trace: https://jaeger.dev.storj.io/trace/32f48b4f1799253 FAIL (2025-11-11 17:11:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ) context deadline exceeded; piecestore: (Node ID: 12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, Piece ID: BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ) context deadline exceeded; piecestore: (Node ID: 12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, Piece ID: XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q) context deadline exceeded; piecestore: (Node ID: 12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, Piece ID: BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA) context deadline exceeded ======LOGS====== 11 17:09:17.808254746 (57h5m16.359725207s) [upload=50295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:09:17.808259335 (57h5m16.359729786s) [upload=50295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:09:18.065943994 (57h5m16.617414455s) [seg_pos={0 0}, upload=50295]: Begin upload segment... Nov 11 17:09:18.066007305 (57h5m16.617477766s) [seg_pos={0 0}, upload=50295]: Done begin upload segment. Nov 11 17:09:18.066048393 (57h5m16.617518854s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.067020474 (57h5m16.618490925s) [noise=true , address=145.79.227.14:4837 , piece_id=XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.067697985 (57h5m16.619168436s) [noise=true , address=45.91.116.11:4118 , piece_id=GLBZIWAUC6OBA3EXQSFDVPQZ3B7AOMA6PSBUJU7GU6CLHRW45NMA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.069137075 (57h5m16.620607536s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.070196713 (57h5m16.621667164s) [noise=true , address=145.79.227.6:4258 , piece_id=BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.074296229 (57h5m16.62576668s) [noise=true , address=45.91.116.4:4940 , piece_id=7HY6FK6RA4AERNEGFIIUNGRKG5UI6QJ6D6NATTFZKKWDSKTGY3PQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.076492341 (57h5m16.627962802s) [noise=true , address=145.79.227.17:4970 , piece_id=EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.078065547 (57h5m16.629536008s) [noise=true , address=145.79.227.9:4365 , piece_id=BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.080540098 (57h5m16.632010549s) [noise=true , address=45.91.116.5:4297 , piece_id=PF7UO5J6C5KHI34EIL4ITP37MSVVNGWW3YC63TJQPWSDNCGIEXYA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.081056011 (57h5m16.632526472s) [noise=true , address=45.91.116.6:4973 , piece_id=DGNIVMHVC476IJ4HVBKEGQQY2ADVWBF5TQ272H3EUBXDXMFRR7OQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.084048509 (57h5m16.63551897s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.085709442 (57h5m16.637179893s) [noise=true , address=77.90.31.12:4592 , piece_id=VRSDWRPJKAWWU3X2DYQSQB5QLTYPHFH6RMIRNSRKQIK75ZF2UAHA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.086524133 (57h5m16.637994594s) [noise=true , address=145.79.227.22:4509 , piece_id=DADDKGVC2EMAEIFHT5GQSQA2BZYWLSW2HFUFDCJWLKTPV5IRPDYQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.087118326 (57h5m16.638588777s) [noise=true , address=45.91.116.13:4153 , piece_id=NK22OZKAZRF746YPBHO27E6QGR2VJI33DAF5TW7FSUMDK7VIU5UQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.088483044 (57h5m16.639953505s) [noise=true , address=77.90.31.15:4697 , piece_id=BCMQWJVGOJ7CDB3WCCTICLSUI7YCO5EONPOB6AHD47M3NSDTW27Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.089429967 (57h5m16.640900428s) [noise=true , address=45.91.116.7:4009 , piece_id=B2ONDGKYMCRX3DSMNSKX34BRJLRXRJD35BKL4FE7FMUFKLGRGERQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:19.843494253 (57h5m18.394964714s) [upload=50295]: Next returned nil segment Nov 11 17:09:19.843513309 (57h5m18.39498379s) [upload=50295]: Waiting for error group managing segments... Nov 11 17:09:20.178022832 (57h5m18.729493333s) [noise=true , address=45.91.116.4:4940 , piece_id=7HY6FK6RA4AERNEGFIIUNGRKG5UI6QJ6D6NATTFZKKWDSKTGY3PQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.191212378 (57h5m18.742682849s) [noise=true , address=77.90.31.12:4592 , piece_id=VRSDWRPJKAWWU3X2DYQSQB5QLTYPHFH6RMIRNSRKQIK75ZF2UAHA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.214652943 (57h5m18.766123394s) [noise=true , address=45.91.116.11:4118 , piece_id=GLBZIWAUC6OBA3EXQSFDVPQZ3B7AOMA6PSBUJU7GU6CLHRW45NMA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.228934699 (57h5m18.78040515s) [noise=true , address=45.91.116.7:4009 , piece_id=B2ONDGKYMCRX3DSMNSKX34BRJLRXRJD35BKL4FE7FMUFKLGRGERQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.257611417 (57h5m18.809081878s) [noise=true , address=45.91.116.6:4973 , piece_id=DGNIVMHVC476IJ4HVBKEGQQY2ADVWBF5TQ272H3EUBXDXMFRR7OQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.286180058 (57h5m18.837650519s) [noise=true , address=45.91.116.13:4153 , piece_id=NK22OZKAZRF746YPBHO27E6QGR2VJI33DAF5TW7FSUMDK7VIU5UQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.361426805 (57h5m18.912897266s) [noise=true , address=45.91.116.5:4297 , piece_id=PF7UO5J6C5KHI34EIL4ITP37MSVVNGWW3YC63TJQPWSDNCGIEXYA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.375330360 (57h5m18.926800821s) [noise=true , address=77.90.31.15:4697 , piece_id=BCMQWJVGOJ7CDB3WCCTICLSUI7YCO5EONPOB6AHD47M3NSDTW27Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:31.518915065 (57h5m30.070385516s) [noise=true , address=145.79.227.22:4509 , piece_id=DADDKGVC2EMAEIFHT5GQSQA2BZYWLSW2HFUFDCJWLKTPV5IRPDYQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:34.758887248 (57h5m33.310357699s) [noise=true , address=145.79.227.9:4365 , piece_id=BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:35.165420831 (57h5m33.716891282s) [noise=true , address=145.79.227.6:4258 , piece_id=BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:35.922385673 (57h5m34.473856124s) [noise=true , address=145.79.227.17:4970 , piece_id=EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:44.840964696 (57h5m43.392435147s) [noise=true , address=145.79.227.14:4837 , piece_id=XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:44.841004482 (57h5m43.392474943s) [seg_pos={0 0}, upload=50295]: Segment reached optimal threshold of 13 pieces. Nov 11 17:09:44.841717200 (57h5m43.393187671s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection Nov 11 17:09:44.841729202 (57h5m43.393199683s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842051686 (57h5m43.393522147s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection Nov 11 17:09:44.842059662 (57h5m43.393530123s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842244383 (57h5m43.393714844s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection Nov 11 17:09:44.842252308 (57h5m43.393722759s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842269892 (57h5m43.393740353s) [seg_pos={0 0}, upload=50295]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 17:09:44.842338914 (57h5m43.393809375s) [upload=50295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 17:09:44.842343502 (57h5m43.393813963s) [upload=50295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 17:09:45.141069414 (57h5m43.692539875s) [upload=50295]: Done uploading segments... Nov 11 17:09:45.141088591 (57h5m43.692559042s) [upload=50295]: Upload finished. err: <nil> Duration: 8.324992045s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [X..............] duration: 49.955597ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/21cc4106a0209460 FAIL (2025-11-11 17:10:51 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:51104->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 49.982358ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.207285982s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 17:10:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.207314627s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................X......X................................XX.] duration: 9.434968684s trace: https://jaeger.dev.storj.io/trace/32f48b4f1799253 FAIL (2025-11-11 17:10:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ) context deadline exceeded; piecestore: (Node ID: 12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, Piece ID: BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ) context deadline exceeded; piecestore: (Node ID: 12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, Piece ID: XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q) context deadline exceeded; piecestore: (Node ID: 12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, Piece ID: BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA) context deadline exceeded ======LOGS====== 11 17:09:17.808254746 (57h5m16.359725207s) [upload=50295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:09:17.808259335 (57h5m16.359729786s) [upload=50295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:09:18.065943994 (57h5m16.617414455s) [seg_pos={0 0}, upload=50295]: Begin upload segment... Nov 11 17:09:18.066007305 (57h5m16.617477766s) [seg_pos={0 0}, upload=50295]: Done begin upload segment. Nov 11 17:09:18.066048393 (57h5m16.617518854s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.067020474 (57h5m16.618490925s) [noise=true , address=145.79.227.14:4837 , piece_id=XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.067697985 (57h5m16.619168436s) [noise=true , address=45.91.116.11:4118 , piece_id=GLBZIWAUC6OBA3EXQSFDVPQZ3B7AOMA6PSBUJU7GU6CLHRW45NMA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.069137075 (57h5m16.620607536s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.070196713 (57h5m16.621667164s) [noise=true , address=145.79.227.6:4258 , piece_id=BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.074296229 (57h5m16.62576668s) [noise=true , address=45.91.116.4:4940 , piece_id=7HY6FK6RA4AERNEGFIIUNGRKG5UI6QJ6D6NATTFZKKWDSKTGY3PQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.076492341 (57h5m16.627962802s) [noise=true , address=145.79.227.17:4970 , piece_id=EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.078065547 (57h5m16.629536008s) [noise=true , address=145.79.227.9:4365 , piece_id=BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.080540098 (57h5m16.632010549s) [noise=true , address=45.91.116.5:4297 , piece_id=PF7UO5J6C5KHI34EIL4ITP37MSVVNGWW3YC63TJQPWSDNCGIEXYA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.081056011 (57h5m16.632526472s) [noise=true , address=45.91.116.6:4973 , piece_id=DGNIVMHVC476IJ4HVBKEGQQY2ADVWBF5TQ272H3EUBXDXMFRR7OQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.084048509 (57h5m16.63551897s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.085709442 (57h5m16.637179893s) [noise=true , address=77.90.31.12:4592 , piece_id=VRSDWRPJKAWWU3X2DYQSQB5QLTYPHFH6RMIRNSRKQIK75ZF2UAHA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.086524133 (57h5m16.637994594s) [noise=true , address=145.79.227.22:4509 , piece_id=DADDKGVC2EMAEIFHT5GQSQA2BZYWLSW2HFUFDCJWLKTPV5IRPDYQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.087118326 (57h5m16.638588777s) [noise=true , address=45.91.116.13:4153 , piece_id=NK22OZKAZRF746YPBHO27E6QGR2VJI33DAF5TW7FSUMDK7VIU5UQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.088483044 (57h5m16.639953505s) [noise=true , address=77.90.31.15:4697 , piece_id=BCMQWJVGOJ7CDB3WCCTICLSUI7YCO5EONPOB6AHD47M3NSDTW27Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.089429967 (57h5m16.640900428s) [noise=true , address=45.91.116.7:4009 , piece_id=B2ONDGKYMCRX3DSMNSKX34BRJLRXRJD35BKL4FE7FMUFKLGRGERQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:19.843494253 (57h5m18.394964714s) [upload=50295]: Next returned nil segment Nov 11 17:09:19.843513309 (57h5m18.39498379s) [upload=50295]: Waiting for error group managing segments... Nov 11 17:09:20.178022832 (57h5m18.729493333s) [noise=true , address=45.91.116.4:4940 , piece_id=7HY6FK6RA4AERNEGFIIUNGRKG5UI6QJ6D6NATTFZKKWDSKTGY3PQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.191212378 (57h5m18.742682849s) [noise=true , address=77.90.31.12:4592 , piece_id=VRSDWRPJKAWWU3X2DYQSQB5QLTYPHFH6RMIRNSRKQIK75ZF2UAHA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.214652943 (57h5m18.766123394s) [noise=true , address=45.91.116.11:4118 , piece_id=GLBZIWAUC6OBA3EXQSFDVPQZ3B7AOMA6PSBUJU7GU6CLHRW45NMA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.228934699 (57h5m18.78040515s) [noise=true , address=45.91.116.7:4009 , piece_id=B2ONDGKYMCRX3DSMNSKX34BRJLRXRJD35BKL4FE7FMUFKLGRGERQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.257611417 (57h5m18.809081878s) [noise=true , address=45.91.116.6:4973 , piece_id=DGNIVMHVC476IJ4HVBKEGQQY2ADVWBF5TQ272H3EUBXDXMFRR7OQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.286180058 (57h5m18.837650519s) [noise=true , address=45.91.116.13:4153 , piece_id=NK22OZKAZRF746YPBHO27E6QGR2VJI33DAF5TW7FSUMDK7VIU5UQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.361426805 (57h5m18.912897266s) [noise=true , address=45.91.116.5:4297 , piece_id=PF7UO5J6C5KHI34EIL4ITP37MSVVNGWW3YC63TJQPWSDNCGIEXYA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.375330360 (57h5m18.926800821s) [noise=true , address=77.90.31.15:4697 , piece_id=BCMQWJVGOJ7CDB3WCCTICLSUI7YCO5EONPOB6AHD47M3NSDTW27Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:31.518915065 (57h5m30.070385516s) [noise=true , address=145.79.227.22:4509 , piece_id=DADDKGVC2EMAEIFHT5GQSQA2BZYWLSW2HFUFDCJWLKTPV5IRPDYQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:34.758887248 (57h5m33.310357699s) [noise=true , address=145.79.227.9:4365 , piece_id=BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:35.165420831 (57h5m33.716891282s) [noise=true , address=145.79.227.6:4258 , piece_id=BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:35.922385673 (57h5m34.473856124s) [noise=true , address=145.79.227.17:4970 , piece_id=EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:44.840964696 (57h5m43.392435147s) [noise=true , address=145.79.227.14:4837 , piece_id=XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:44.841004482 (57h5m43.392474943s) [seg_pos={0 0}, upload=50295]: Segment reached optimal threshold of 13 pieces. Nov 11 17:09:44.841717200 (57h5m43.393187671s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection Nov 11 17:09:44.841729202 (57h5m43.393199683s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842051686 (57h5m43.393522147s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection Nov 11 17:09:44.842059662 (57h5m43.393530123s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842244383 (57h5m43.393714844s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection Nov 11 17:09:44.842252308 (57h5m43.393722759s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842269892 (57h5m43.393740353s) [seg_pos={0 0}, upload=50295]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 17:09:44.842338914 (57h5m43.393809375s) [upload=50295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 17:09:44.842343502 (57h5m43.393813963s) [upload=50295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 17:09:45.141069414 (57h5m43.692539875s) [upload=50295]: Done uploading segments... Nov 11 17:09:45.141088591 (57h5m43.692559042s) [upload=50295]: Upload finished. err: <nil> Duration: 9.434992511s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.625204389s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 17:10:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.62523598s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.X.............] duration: 53.081568ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/21cc4106a0209460 FAIL (2025-11-11 17:09:48 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:51104->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 53.100686ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................X......X................................XX] duration: 30.005421389s trace: https://jaeger.dev.storj.io/trace/32f48b4f1799253 FAIL (2025-11-11 17:09:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ) context deadline exceeded; piecestore: (Node ID: 12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, Piece ID: BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ) context deadline exceeded; piecestore: (Node ID: 12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, Piece ID: XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q) context deadline exceeded; piecestore: (Node ID: 12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, Piece ID: BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA) context deadline exceeded ======LOGS====== 11 17:09:17.808254746 (57h5m16.359725207s) [upload=50295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:09:17.808259335 (57h5m16.359729786s) [upload=50295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:09:18.065943994 (57h5m16.617414455s) [seg_pos={0 0}, upload=50295]: Begin upload segment... Nov 11 17:09:18.066007305 (57h5m16.617477766s) [seg_pos={0 0}, upload=50295]: Done begin upload segment. Nov 11 17:09:18.066048393 (57h5m16.617518854s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.067020474 (57h5m16.618490925s) [noise=true , address=145.79.227.14:4837 , piece_id=XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.067697985 (57h5m16.619168436s) [noise=true , address=45.91.116.11:4118 , piece_id=GLBZIWAUC6OBA3EXQSFDVPQZ3B7AOMA6PSBUJU7GU6CLHRW45NMA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.069137075 (57h5m16.620607536s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.070196713 (57h5m16.621667164s) [noise=true , address=145.79.227.6:4258 , piece_id=BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.074296229 (57h5m16.62576668s) [noise=true , address=45.91.116.4:4940 , piece_id=7HY6FK6RA4AERNEGFIIUNGRKG5UI6QJ6D6NATTFZKKWDSKTGY3PQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.076492341 (57h5m16.627962802s) [noise=true , address=145.79.227.17:4970 , piece_id=EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.078065547 (57h5m16.629536008s) [noise=true , address=145.79.227.9:4365 , piece_id=BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.080540098 (57h5m16.632010549s) [noise=true , address=45.91.116.5:4297 , piece_id=PF7UO5J6C5KHI34EIL4ITP37MSVVNGWW3YC63TJQPWSDNCGIEXYA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.081056011 (57h5m16.632526472s) [noise=true , address=45.91.116.6:4973 , piece_id=DGNIVMHVC476IJ4HVBKEGQQY2ADVWBF5TQ272H3EUBXDXMFRR7OQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.084048509 (57h5m16.63551897s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.085709442 (57h5m16.637179893s) [noise=true , address=77.90.31.12:4592 , piece_id=VRSDWRPJKAWWU3X2DYQSQB5QLTYPHFH6RMIRNSRKQIK75ZF2UAHA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.086524133 (57h5m16.637994594s) [noise=true , address=145.79.227.22:4509 , piece_id=DADDKGVC2EMAEIFHT5GQSQA2BZYWLSW2HFUFDCJWLKTPV5IRPDYQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.087118326 (57h5m16.638588777s) [noise=true , address=45.91.116.13:4153 , piece_id=NK22OZKAZRF746YPBHO27E6QGR2VJI33DAF5TW7FSUMDK7VIU5UQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.088483044 (57h5m16.639953505s) [noise=true , address=77.90.31.15:4697 , piece_id=BCMQWJVGOJ7CDB3WCCTICLSUI7YCO5EONPOB6AHD47M3NSDTW27Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.089429967 (57h5m16.640900428s) [noise=true , address=45.91.116.7:4009 , piece_id=B2ONDGKYMCRX3DSMNSKX34BRJLRXRJD35BKL4FE7FMUFKLGRGERQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:19.843494253 (57h5m18.394964714s) [upload=50295]: Next returned nil segment Nov 11 17:09:19.843513309 (57h5m18.39498379s) [upload=50295]: Waiting for error group managing segments... Nov 11 17:09:20.178022832 (57h5m18.729493333s) [noise=true , address=45.91.116.4:4940 , piece_id=7HY6FK6RA4AERNEGFIIUNGRKG5UI6QJ6D6NATTFZKKWDSKTGY3PQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.191212378 (57h5m18.742682849s) [noise=true , address=77.90.31.12:4592 , piece_id=VRSDWRPJKAWWU3X2DYQSQB5QLTYPHFH6RMIRNSRKQIK75ZF2UAHA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.214652943 (57h5m18.766123394s) [noise=true , address=45.91.116.11:4118 , piece_id=GLBZIWAUC6OBA3EXQSFDVPQZ3B7AOMA6PSBUJU7GU6CLHRW45NMA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.228934699 (57h5m18.78040515s) [noise=true , address=45.91.116.7:4009 , piece_id=B2ONDGKYMCRX3DSMNSKX34BRJLRXRJD35BKL4FE7FMUFKLGRGERQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.257611417 (57h5m18.809081878s) [noise=true , address=45.91.116.6:4973 , piece_id=DGNIVMHVC476IJ4HVBKEGQQY2ADVWBF5TQ272H3EUBXDXMFRR7OQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.286180058 (57h5m18.837650519s) [noise=true , address=45.91.116.13:4153 , piece_id=NK22OZKAZRF746YPBHO27E6QGR2VJI33DAF5TW7FSUMDK7VIU5UQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.361426805 (57h5m18.912897266s) [noise=true , address=45.91.116.5:4297 , piece_id=PF7UO5J6C5KHI34EIL4ITP37MSVVNGWW3YC63TJQPWSDNCGIEXYA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.375330360 (57h5m18.926800821s) [noise=true , address=77.90.31.15:4697 , piece_id=BCMQWJVGOJ7CDB3WCCTICLSUI7YCO5EONPOB6AHD47M3NSDTW27Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:31.518915065 (57h5m30.070385516s) [noise=true , address=145.79.227.22:4509 , piece_id=DADDKGVC2EMAEIFHT5GQSQA2BZYWLSW2HFUFDCJWLKTPV5IRPDYQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:34.758887248 (57h5m33.310357699s) [noise=true , address=145.79.227.9:4365 , piece_id=BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:35.165420831 (57h5m33.716891282s) [noise=true , address=145.79.227.6:4258 , piece_id=BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:35.922385673 (57h5m34.473856124s) [noise=true , address=145.79.227.17:4970 , piece_id=EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:44.840964696 (57h5m43.392435147s) [noise=true , address=145.79.227.14:4837 , piece_id=XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:44.841004482 (57h5m43.392474943s) [seg_pos={0 0}, upload=50295]: Segment reached optimal threshold of 13 pieces. Nov 11 17:09:44.841717200 (57h5m43.393187671s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection Nov 11 17:09:44.841729202 (57h5m43.393199683s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842051686 (57h5m43.393522147s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection Nov 11 17:09:44.842059662 (57h5m43.393530123s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842244383 (57h5m43.393714844s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection Nov 11 17:09:44.842252308 (57h5m43.393722759s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842269892 (57h5m43.393740353s) [seg_pos={0 0}, upload=50295]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 17:09:44.842338914 (57h5m43.393809375s) [upload=50295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 17:09:44.842343502 (57h5m43.393813963s) [upload=50295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 17:09:45.141069414 (57h5m43.692539875s) [upload=50295]: Done uploading segments... Nov 11 17:09:45.141088591 (57h5m43.692559042s) [upload=50295]: Upload finished. err: <nil> Duration: 30.005446016s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 1.444307019s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 17:09:47 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.44432766s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 9.501005785s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 17:09:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 9.501042856s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..X............] duration: 48.76987ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/21cc4106a0209460 FAIL (2025-11-11 17:09:32 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:51104->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 48.800737ms core website
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 1.436167659s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 17:09:21 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.436197466s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X......X................................X] duration: 30.252166264s trace: https://jaeger.dev.storj.io/trace/371b4b79a1c06bce FAIL (2025-11-11 17:09:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== pos={0 0}, upload=50286]: Uploading piece... Nov 11 17:08:45.547361428 (57h4m44.098831879s) [noise=true , address=145.79.227.32:4836 , piece_id=IAFB6VITPF5DCIJZRLH3ND7P7J5NYZ56L4DOKGVFWUQGSRPJOV2Q, seg_pos={0 0}, upload=50286]: Uploading piece... Nov 11 17:08:45.547391596 (57h4m44.098862057s) [noise=true , address=77.90.31.11:4553 , piece_id=UV42S5AZVTSWQGCCBW34364PQX5XIWORAJ23AQXVGWPDW6OQFUKQ, seg_pos={0 0}, upload=50286]: Uploading piece... Nov 11 17:08:45.547726995 (57h4m44.099197446s) [noise=true , address=145.79.227.15:4874 , piece_id=G4EK43PVEZCA75HE4Z2KDNEVJ2FW2ED6I2XHCI7R2434N24H7M7Q, seg_pos={0 0}, upload=50286]: Uploading piece... Nov 11 17:08:45.547841132 (57h4m44.099311593s) [noise=true , address=145.79.227.13:4802 , piece_id=KSC4CDNHDTA6CJ7MAPNKLUCRJCZ6A3DIJWBWWOKDVYAGLEORCIPQ, seg_pos={0 0}, upload=50286]: Uploading piece... Nov 11 17:08:45.548096758 (57h4m44.099567209s) [noise=true , address=145.79.227.26:4653 , piece_id=QL4SARHCZCQPHPMC7H2TVSJMU454M5OLTBFAAONA7NMGZUXBIK5A, seg_pos={0 0}, upload=50286]: Uploading piece... Nov 11 17:08:45.548298262 (57h4m44.099768723s) [noise=true , address=77.90.31.17:4771 , piece_id=MFAZQJ2D3QFP72CTMWQA65B2R2QUJGOQFYZ3UGTOTW7X3WTJ56ZQ, seg_pos={0 0}, upload=50286]: Uploading piece... Nov 11 17:08:45.548584918 (57h4m44.100055369s) [noise=true , address=145.79.227.22:4511 , piece_id=EHU4OHFYWAHB3D6U7CBBA5JVLHITPVHWF6SKEZVQTTIGJDF5DM4A, seg_pos={0 0}, upload=50286]: Uploading piece... Nov 11 17:08:45.549095060 (57h4m44.100565521s) [noise=true , address=145.79.227.14:4840 , piece_id=A5UKZEZHHXGLQXVUY3EC76PTCQSUYWIAZYDLBK3SLJMEFMPDKM7A, seg_pos={0 0}, upload=50286]: Uploading piece... Nov 11 17:08:45.549725450 (57h4m44.101195911s) [noise=true , address=145.79.227.17:4970 , piece_id=Y6MYJ6JDKCWOXNOAAU6VIZ2SFLUG6JLG6I7L4AEH2DL6ZKKPBZUQ, seg_pos={0 0}, upload=50286]: Uploading piece... Nov 11 17:08:45.550230562 (57h4m44.101701023s) [noise=true , address=145.79.227.25:4617 , piece_id=6YBJD7DPCTN5SOHXJVCUYKTPM3VXVAKRZOUOM2AKYXBX4N2VS4KQ, seg_pos={0 0}, upload=50286]: Uploading piece... Nov 11 17:08:45.550716427 (57h4m44.102186888s) [noise=true , address=77.90.31.13:4627 , piece_id=FGXY44XEJTEHSVCOSKBKI7UHIMDGXMR42MUWEAD6N723BSIETXMA, seg_pos={0 0}, upload=50286]: Uploading piece... Nov 11 17:08:45.551603967 (57h4m44.103074438s) [noise=true , address=45.91.116.14:4191 , piece_id=PBQK5R75P3RFLLUBXFA6ZW6JC3M743GRJXUD3MOATB22QBZF5P7Q, seg_pos={0 0}, upload=50286]: Uploading piece... Nov 11 17:08:45.552148855 (57h4m44.103619326s) [noise=true , address=77.90.31.18:4805 , piece_id=EGX7OZ4BI3O5NEXXXIZNO7YLZQVYKUPIHIZ5FYAQTCJVUWP3HJJA, seg_pos={0 0}, upload=50286]: Uploading piece... Nov 11 17:08:47.321097644 (57h4m45.872568105s) [upload=50286]: Next returned nil segment Nov 11 17:08:47.321130206 (57h4m45.872600667s) [upload=50286]: Waiting for error group managing segments... Nov 11 17:08:47.603824170 (57h4m46.155294631s) [noise=true , address=77.90.31.21:4877 , piece_id=OMCJJW55HN7WVDSGX4FCYF4464EVD6UN4C2GRJ3SSJC2575QWV6A, seg_pos={0 0}, upload=50286]: Done uploading piece. err: <nil> Nov 11 17:08:47.655805575 (57h4m46.207276026s) [noise=true , address=45.91.116.14:4191 , piece_id=PBQK5R75P3RFLLUBXFA6ZW6JC3M743GRJXUD3MOATB22QBZF5P7Q, seg_pos={0 0}, upload=50286]: Done uploading piece. err: <nil> Nov 11 17:08:47.657923087 (57h4m46.209393548s) [noise=true , address=45.91.116.11:4118 , piece_id=C63TYBCOLXE6NJJKXMA5HAFT647DXGZX5I7XNP4LVK4RHZVLIMZA, seg_pos={0 0}, upload=50286]: Done uploading piece. err: <nil> Nov 11 17:08:47.661155502 (57h4m46.212625963s) [noise=true , address=77.90.31.18:4805 , piece_id=EGX7OZ4BI3O5NEXXXIZNO7YLZQVYKUPIHIZ5FYAQTCJVUWP3HJJA, seg_pos={0 0}, upload=50286]: Done uploading piece. err: <nil> Nov 11 17:08:47.704832433 (57h4m46.256302884s) [noise=true , address=77.90.31.13:4627 , piece_id=FGXY44XEJTEHSVCOSKBKI7UHIMDGXMR42MUWEAD6N723BSIETXMA, seg_pos={0 0}, upload=50286]: Done uploading piece. err: <nil> Nov 11 17:08:47.715739533 (57h4m46.267209984s) [noise=true , address=77.90.31.17:4771 , piece_id=MFAZQJ2D3QFP72CTMWQA65B2R2QUJGOQFYZ3UGTOTW7X3WTJ56ZQ, seg_pos={0 0}, upload=50286]: Done uploading piece. err: <nil> Nov 11 17:08:47.742043181 (57h4m46.293513642s) [noise=true , address=77.90.31.11:4553 , piece_id=UV42S5AZVTSWQGCCBW34364PQX5XIWORAJ23AQXVGWPDW6OQFUKQ, seg_pos={0 0}, upload=50286]: Done uploading piece. err: <nil> Nov 11 17:09:02.278627798 (57h5m0.830098259s) [noise=true , address=145.79.227.25:4617 , piece_id=6YBJD7DPCTN5SOHXJVCUYKTPM3VXVAKRZOUOM2AKYXBX4N2VS4KQ, seg_pos={0 0}, upload=50286]: Done uploading piece. err: <nil> Nov 11 17:09:14.561473016 (57h5m13.112943477s) [noise=true , address=145.79.227.22:4511 , piece_id=EHU4OHFYWAHB3D6U7CBBA5JVLHITPVHWF6SKEZVQTTIGJDF5DM4A, seg_pos={0 0}, upload=50286]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:43610->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:43610->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43610->145.79.227.22:4511: use of closed network connection Nov 11 17:09:14.561633501 (57h5m13.113103962s) [noise=true , address=145.79.227.32:4836 , piece_id=IAFB6VITPF5DCIJZRLH3ND7P7J5NYZ56L4DOKGVFWUQGSRPJOV2Q, seg_pos={0 0}, upload=50286]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:33940->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:33940->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33940->145.79.227.32:4836: use of closed network connection Nov 11 17:09:14.561741737 (57h5m13.113212188s) [noise=true , address=145.79.227.13:4802 , piece_id=KSC4CDNHDTA6CJ7MAPNKLUCRJCZ6A3DIJWBWWOKDVYAGLEORCIPQ, seg_pos={0 0}, upload=50286]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:45344->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:45344->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45344->145.79.227.13:4802: use of closed network connection Nov 11 17:09:14.561912121 (57h5m13.113382583s) [noise=true , address=145.79.227.17:4970 , piece_id=Y6MYJ6JDKCWOXNOAAU6VIZ2SFLUG6JLG6I7L4AEH2DL6ZKKPBZUQ, seg_pos={0 0}, upload=50286]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:39424->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:39424->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39424->145.79.227.17:4970: use of closed network connection Nov 11 17:09:14.562017031 (57h5m13.113487482s) [noise=true , address=145.79.227.9:4365 , piece_id=I57DY66ZWYWSX4DXLV4Y7N7LFMEZYK2LEZ5TFQXJ7YOLERTGREAA, seg_pos={0 0}, upload=50286]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:35968->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:35968->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35968->145.79.227.9:4365: use of closed network connection Nov 11 17:09:14.562147380 (57h5m13.113617841s) [noise=true , address=145.79.227.14:4840 , piece_id=A5UKZEZHHXGLQXVUY3EC76PTCQSUYWIAZYDLBK3SLJMEFMPDKM7A, seg_pos={0 0}, upload=50286]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:33558->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:33558->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33558->145.79.227.14:4840: use of closed network connection Nov 11 17:09:14.562271877 (57h5m13.113742338s) [noise=true , address=145.79.227.15:4874 , piece_id=G4EK43PVEZCA75HE4Z2KDNEVJ2FW2ED6I2XHCI7R2434N24H7M7Q, seg_pos={0 0}, upload=50286]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:54878->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:54878->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54878->145.79.227.15:4874: use of closed network connection Nov 11 17:09:14.562798510 (57h5m13.114268961s) [noise=true , address=145.79.227.26:4653 , piece_id=QL4SARHCZCQPHPMC7H2TVSJMU454M5OLTBFAAONA7NMGZUXBIK5A, seg_pos={0 0}, upload=50286]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:49332->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:49332->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49332->145.79.227.26:4653: use of closed network connection Nov 11 17:09:14.562841301 (57h5m13.114311762s) [seg_pos={0 0}, upload=50286]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:09:14.811679715 (57h5m13.363150176s) [upload=50286]: Done uploading segments... Nov 11 17:09:14.811723850 (57h5m13.363194311s) [upload=50286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.252221519s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.538257997s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 17:08:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.53829141s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [...X...........] duration: 76.590154ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/21cc4106a0209460 FAIL (2025-11-11 17:08:28 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:51104->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 76.62949ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X......X................................] duration: 7.761980542s trace: https://jaeger.dev.storj.io/trace/406600cd16680b82 FAIL (2025-11-11 17:08:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 952]: Done starting upload Nov 11 16:45:40.082304163 (56h41m38.633774624s) [upload=49952]: Uploading segments... Nov 11 16:45:40.082345682 (56h41m38.633816133s) [upload=49952]: Got next segment. Inline: false Nov 11 16:45:40.082372463 (56h41m38.633842924s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 16:45:40.082377623 (56h41m38.633848074s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 16:45:40.343320114 (56h41m38.894790575s) [seg_pos={0 0}, upload=49952]: Begin upload segment... Nov 11 16:45:40.343375529 (56h41m38.89484599s) [seg_pos={0 0}, upload=49952]: Done begin upload segment. Nov 11 16:45:40.343423430 (56h41m38.894893881s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.343448578 (56h41m38.894919029s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344383769 (56h41m38.89585423s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344997518 (56h41m38.896467979s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345095395 (56h41m38.896565856s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345501417 (56h41m38.896971868s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346057606 (56h41m38.897528067s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346422612 (56h41m38.897893063s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346447920 (56h41m38.897918381s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346801393 (56h41m38.898271854s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347099851 (56h41m38.898570302s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347169985 (56h41m38.898640446s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347569877 (56h41m38.899040328s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347577421 (56h41m38.899047872s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347916837 (56h41m38.899387298s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.348105827 (56h41m38.899576288s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:42.155239479 (56h41m40.70670994s) [upload=49952]: Next returned nil segment Nov 11 16:45:42.155253296 (56h41m40.706723757s) [upload=49952]: Waiting for error group managing segments... Nov 11 16:45:42.509736187 (56h41m41.061206668s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.523781983 (56h41m41.075252444s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.524977681 (56h41m41.076448162s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.544084193 (56h41m41.095554654s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.545880774 (56h41m41.097351236s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.617572702 (56h41m41.169043163s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:49.538637614 (56h41m48.090108075s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.209224227 (56h41m53.760694688s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.376479046 (56h41m53.927949507s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.381095176 (56h41m53.932565637s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:02.516152664 (56h42m1.067623125s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:07.807035175 (56h42m6.358505636s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165478509 (56h42m6.71694897s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165504899 (56h42m6.71697536s) [seg_pos={0 0}, upload=49952]: Segment reached optimal threshold of 13 pieces. Nov 11 16:46:08.165731340 (56h42m6.717201801s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection Nov 11 16:46:08.165742050 (56h42m6.717212521s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.165752790 (56h42m6.717223252s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection Nov 11 16:46:08.165760676 (56h42m6.717231137s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166125290 (56h42m6.717595751s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Nov 11 16:46:08.166135960 (56h42m6.717606421s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166155798 (56h42m6.717626259s) [seg_pos={0 0}, upload=49952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 16:46:08.166239006 (56h42m6.717709467s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 16:46:08.166245079 (56h42m6.71771553s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 16:46:08.467714732 (56h42m7.019185193s) [upload=49952]: Done uploading segments... Nov 11 16:46:08.467737616 (56h42m7.019208077s) [upload=49952]: Upload finished. err: <nil> Duration: 7.762175202s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 1.460360872s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 17:08:17 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.460394846s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X......X...............................] duration: 8.166228811s trace: https://jaeger.dev.storj.io/trace/406600cd16680b82 FAIL (2025-11-11 17:08:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 952]: Done starting upload Nov 11 16:45:40.082304163 (56h41m38.633774624s) [upload=49952]: Uploading segments... Nov 11 16:45:40.082345682 (56h41m38.633816133s) [upload=49952]: Got next segment. Inline: false Nov 11 16:45:40.082372463 (56h41m38.633842924s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 16:45:40.082377623 (56h41m38.633848074s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 16:45:40.343320114 (56h41m38.894790575s) [seg_pos={0 0}, upload=49952]: Begin upload segment... Nov 11 16:45:40.343375529 (56h41m38.89484599s) [seg_pos={0 0}, upload=49952]: Done begin upload segment. Nov 11 16:45:40.343423430 (56h41m38.894893881s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.343448578 (56h41m38.894919029s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344383769 (56h41m38.89585423s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344997518 (56h41m38.896467979s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345095395 (56h41m38.896565856s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345501417 (56h41m38.896971868s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346057606 (56h41m38.897528067s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346422612 (56h41m38.897893063s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346447920 (56h41m38.897918381s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346801393 (56h41m38.898271854s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347099851 (56h41m38.898570302s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347169985 (56h41m38.898640446s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347569877 (56h41m38.899040328s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347577421 (56h41m38.899047872s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347916837 (56h41m38.899387298s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.348105827 (56h41m38.899576288s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:42.155239479 (56h41m40.70670994s) [upload=49952]: Next returned nil segment Nov 11 16:45:42.155253296 (56h41m40.706723757s) [upload=49952]: Waiting for error group managing segments... Nov 11 16:45:42.509736187 (56h41m41.061206668s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.523781983 (56h41m41.075252444s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.524977681 (56h41m41.076448162s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.544084193 (56h41m41.095554654s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.545880774 (56h41m41.097351236s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.617572702 (56h41m41.169043163s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:49.538637614 (56h41m48.090108075s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.209224227 (56h41m53.760694688s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.376479046 (56h41m53.927949507s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.381095176 (56h41m53.932565637s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:02.516152664 (56h42m1.067623125s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:07.807035175 (56h42m6.358505636s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165478509 (56h42m6.71694897s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165504899 (56h42m6.71697536s) [seg_pos={0 0}, upload=49952]: Segment reached optimal threshold of 13 pieces. Nov 11 16:46:08.165731340 (56h42m6.717201801s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection Nov 11 16:46:08.165742050 (56h42m6.717212521s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.165752790 (56h42m6.717223252s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection Nov 11 16:46:08.165760676 (56h42m6.717231137s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166125290 (56h42m6.717595751s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Nov 11 16:46:08.166135960 (56h42m6.717606421s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166155798 (56h42m6.717626259s) [seg_pos={0 0}, upload=49952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 16:46:08.166239006 (56h42m6.717709467s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 16:46:08.166245079 (56h42m6.71771553s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 16:46:08.467714732 (56h42m7.019185193s) [upload=49952]: Done uploading segments... Nov 11 16:46:08.467737616 (56h42m7.019208077s) [upload=49952]: Upload finished. err: <nil> Duration: 8.166260401s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 1.477984509s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 17:07:55 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.47801204s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 1.444744691s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 17:07:35 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.444775851s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.310173963s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 17:07:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.310213228s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [....X..........] duration: 68.508405ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/21cc4106a0209460 FAIL (2025-11-11 17:07:19 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:51104->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 68.536408ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X......X..............................] duration: 9.189118568s trace: https://jaeger.dev.storj.io/trace/406600cd16680b82 FAIL (2025-11-11 17:07:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 952]: Done starting upload Nov 11 16:45:40.082304163 (56h41m38.633774624s) [upload=49952]: Uploading segments... Nov 11 16:45:40.082345682 (56h41m38.633816133s) [upload=49952]: Got next segment. Inline: false Nov 11 16:45:40.082372463 (56h41m38.633842924s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 16:45:40.082377623 (56h41m38.633848074s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 16:45:40.343320114 (56h41m38.894790575s) [seg_pos={0 0}, upload=49952]: Begin upload segment... Nov 11 16:45:40.343375529 (56h41m38.89484599s) [seg_pos={0 0}, upload=49952]: Done begin upload segment. Nov 11 16:45:40.343423430 (56h41m38.894893881s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.343448578 (56h41m38.894919029s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344383769 (56h41m38.89585423s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344997518 (56h41m38.896467979s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345095395 (56h41m38.896565856s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345501417 (56h41m38.896971868s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346057606 (56h41m38.897528067s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346422612 (56h41m38.897893063s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346447920 (56h41m38.897918381s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346801393 (56h41m38.898271854s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347099851 (56h41m38.898570302s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347169985 (56h41m38.898640446s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347569877 (56h41m38.899040328s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347577421 (56h41m38.899047872s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347916837 (56h41m38.899387298s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.348105827 (56h41m38.899576288s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:42.155239479 (56h41m40.70670994s) [upload=49952]: Next returned nil segment Nov 11 16:45:42.155253296 (56h41m40.706723757s) [upload=49952]: Waiting for error group managing segments... Nov 11 16:45:42.509736187 (56h41m41.061206668s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.523781983 (56h41m41.075252444s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.524977681 (56h41m41.076448162s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.544084193 (56h41m41.095554654s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.545880774 (56h41m41.097351236s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.617572702 (56h41m41.169043163s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:49.538637614 (56h41m48.090108075s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.209224227 (56h41m53.760694688s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.376479046 (56h41m53.927949507s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.381095176 (56h41m53.932565637s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:02.516152664 (56h42m1.067623125s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:07.807035175 (56h42m6.358505636s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165478509 (56h42m6.71694897s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165504899 (56h42m6.71697536s) [seg_pos={0 0}, upload=49952]: Segment reached optimal threshold of 13 pieces. Nov 11 16:46:08.165731340 (56h42m6.717201801s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection Nov 11 16:46:08.165742050 (56h42m6.717212521s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.165752790 (56h42m6.717223252s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection Nov 11 16:46:08.165760676 (56h42m6.717231137s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166125290 (56h42m6.717595751s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Nov 11 16:46:08.166135960 (56h42m6.717606421s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166155798 (56h42m6.717626259s) [seg_pos={0 0}, upload=49952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 16:46:08.166239006 (56h42m6.717709467s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 16:46:08.166245079 (56h42m6.71771553s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 16:46:08.467714732 (56h42m7.019185193s) [upload=49952]: Done uploading segments... Nov 11 16:46:08.467737616 (56h42m7.019208077s) [upload=49952]: Upload finished. err: <nil> Duration: 9.189148425s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.703930619s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 17:06:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.703961406s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 1.431273759s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 17:06:52 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.431307093s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.925212295s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 17:06:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.925244506s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.....X.........] duration: 57.305342ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/21cc4106a0209460 FAIL (2025-11-11 17:06:19 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:51104->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 57.338505ms core website
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 1.45052035s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 17:06:15 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.450539817s core uplink eu1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [X..............] duration: 303.295844ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/553c0de2a794f216 FAIL (2025-11-11 17:06:13 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:57188->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 303.322654ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X......X.............................] duration: 7.513978606s trace: https://jaeger.dev.storj.io/trace/406600cd16680b82 FAIL (2025-11-11 17:06:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 952]: Done starting upload Nov 11 16:45:40.082304163 (56h41m38.633774624s) [upload=49952]: Uploading segments... Nov 11 16:45:40.082345682 (56h41m38.633816133s) [upload=49952]: Got next segment. Inline: false Nov 11 16:45:40.082372463 (56h41m38.633842924s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 16:45:40.082377623 (56h41m38.633848074s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 16:45:40.343320114 (56h41m38.894790575s) [seg_pos={0 0}, upload=49952]: Begin upload segment... Nov 11 16:45:40.343375529 (56h41m38.89484599s) [seg_pos={0 0}, upload=49952]: Done begin upload segment. Nov 11 16:45:40.343423430 (56h41m38.894893881s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.343448578 (56h41m38.894919029s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344383769 (56h41m38.89585423s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344997518 (56h41m38.896467979s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345095395 (56h41m38.896565856s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345501417 (56h41m38.896971868s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346057606 (56h41m38.897528067s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346422612 (56h41m38.897893063s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346447920 (56h41m38.897918381s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346801393 (56h41m38.898271854s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347099851 (56h41m38.898570302s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347169985 (56h41m38.898640446s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347569877 (56h41m38.899040328s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347577421 (56h41m38.899047872s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347916837 (56h41m38.899387298s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.348105827 (56h41m38.899576288s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:42.155239479 (56h41m40.70670994s) [upload=49952]: Next returned nil segment Nov 11 16:45:42.155253296 (56h41m40.706723757s) [upload=49952]: Waiting for error group managing segments... Nov 11 16:45:42.509736187 (56h41m41.061206668s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.523781983 (56h41m41.075252444s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.524977681 (56h41m41.076448162s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.544084193 (56h41m41.095554654s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.545880774 (56h41m41.097351236s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.617572702 (56h41m41.169043163s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:49.538637614 (56h41m48.090108075s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.209224227 (56h41m53.760694688s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.376479046 (56h41m53.927949507s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.381095176 (56h41m53.932565637s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:02.516152664 (56h42m1.067623125s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:07.807035175 (56h42m6.358505636s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165478509 (56h42m6.71694897s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165504899 (56h42m6.71697536s) [seg_pos={0 0}, upload=49952]: Segment reached optimal threshold of 13 pieces. Nov 11 16:46:08.165731340 (56h42m6.717201801s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection Nov 11 16:46:08.165742050 (56h42m6.717212521s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.165752790 (56h42m6.717223252s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection Nov 11 16:46:08.165760676 (56h42m6.717231137s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166125290 (56h42m6.717595751s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Nov 11 16:46:08.166135960 (56h42m6.717606421s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166155798 (56h42m6.717626259s) [seg_pos={0 0}, upload=49952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 16:46:08.166239006 (56h42m6.717709467s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 16:46:08.166245079 (56h42m6.71771553s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 16:46:08.467714732 (56h42m7.019185193s) [upload=49952]: Done uploading segments... Nov 11 16:46:08.467737616 (56h42m7.019208077s) [upload=49952]: Upload finished. err: <nil> Duration: 7.514013483s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 1.500943739s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 17:05:43 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.50096554s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X......X............................] duration: 9.085708749s trace: https://jaeger.dev.storj.io/trace/406600cd16680b82 FAIL (2025-11-11 17:05:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 952]: Done starting upload Nov 11 16:45:40.082304163 (56h41m38.633774624s) [upload=49952]: Uploading segments... Nov 11 16:45:40.082345682 (56h41m38.633816133s) [upload=49952]: Got next segment. Inline: false Nov 11 16:45:40.082372463 (56h41m38.633842924s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 16:45:40.082377623 (56h41m38.633848074s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 16:45:40.343320114 (56h41m38.894790575s) [seg_pos={0 0}, upload=49952]: Begin upload segment... Nov 11 16:45:40.343375529 (56h41m38.89484599s) [seg_pos={0 0}, upload=49952]: Done begin upload segment. Nov 11 16:45:40.343423430 (56h41m38.894893881s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.343448578 (56h41m38.894919029s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344383769 (56h41m38.89585423s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344997518 (56h41m38.896467979s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345095395 (56h41m38.896565856s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345501417 (56h41m38.896971868s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346057606 (56h41m38.897528067s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346422612 (56h41m38.897893063s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346447920 (56h41m38.897918381s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346801393 (56h41m38.898271854s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347099851 (56h41m38.898570302s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347169985 (56h41m38.898640446s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347569877 (56h41m38.899040328s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347577421 (56h41m38.899047872s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347916837 (56h41m38.899387298s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.348105827 (56h41m38.899576288s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:42.155239479 (56h41m40.70670994s) [upload=49952]: Next returned nil segment Nov 11 16:45:42.155253296 (56h41m40.706723757s) [upload=49952]: Waiting for error group managing segments... Nov 11 16:45:42.509736187 (56h41m41.061206668s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.523781983 (56h41m41.075252444s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.524977681 (56h41m41.076448162s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.544084193 (56h41m41.095554654s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.545880774 (56h41m41.097351236s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.617572702 (56h41m41.169043163s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:49.538637614 (56h41m48.090108075s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.209224227 (56h41m53.760694688s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.376479046 (56h41m53.927949507s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.381095176 (56h41m53.932565637s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:02.516152664 (56h42m1.067623125s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:07.807035175 (56h42m6.358505636s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165478509 (56h42m6.71694897s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165504899 (56h42m6.71697536s) [seg_pos={0 0}, upload=49952]: Segment reached optimal threshold of 13 pieces. Nov 11 16:46:08.165731340 (56h42m6.717201801s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection Nov 11 16:46:08.165742050 (56h42m6.717212521s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.165752790 (56h42m6.717223252s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection Nov 11 16:46:08.165760676 (56h42m6.717231137s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166125290 (56h42m6.717595751s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Nov 11 16:46:08.166135960 (56h42m6.717606421s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166155798 (56h42m6.717626259s) [seg_pos={0 0}, upload=49952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 16:46:08.166239006 (56h42m6.717709467s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 16:46:08.166245079 (56h42m6.71771553s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 16:46:08.467714732 (56h42m7.019185193s) [upload=49952]: Done uploading segments... Nov 11 16:46:08.467737616 (56h42m7.019208077s) [upload=49952]: Upload finished. err: <nil> Duration: 9.08574171s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [......X........] duration: 53.028979ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/21cc4106a0209460 FAIL (2025-11-11 17:05:33 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:51104->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 53.058025ms core website
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 1.431901274s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 17:05:26 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.431927525s core uplink eu1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.X.............] duration: 309.950517ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/553c0de2a794f216 FAIL (2025-11-11 17:05:19 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:57188->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 309.980564ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.67037953s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 17:05:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.670408724s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.208681968s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 17:05:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.208717758s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.......X.......] duration: 67.66542ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/21cc4106a0209460 FAIL (2025-11-11 17:04:58 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:51104->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 67.683906ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [..X............] duration: 314.398956ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/553c0de2a794f216 FAIL (2025-11-11 17:04:51 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:57188->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 314.428653ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X......X...........................] duration: 8.533669505s trace: https://jaeger.dev.storj.io/trace/406600cd16680b82 FAIL (2025-11-11 17:04:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 952]: Done starting upload Nov 11 16:45:40.082304163 (56h41m38.633774624s) [upload=49952]: Uploading segments... Nov 11 16:45:40.082345682 (56h41m38.633816133s) [upload=49952]: Got next segment. Inline: false Nov 11 16:45:40.082372463 (56h41m38.633842924s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 16:45:40.082377623 (56h41m38.633848074s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 16:45:40.343320114 (56h41m38.894790575s) [seg_pos={0 0}, upload=49952]: Begin upload segment... Nov 11 16:45:40.343375529 (56h41m38.89484599s) [seg_pos={0 0}, upload=49952]: Done begin upload segment. Nov 11 16:45:40.343423430 (56h41m38.894893881s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.343448578 (56h41m38.894919029s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344383769 (56h41m38.89585423s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344997518 (56h41m38.896467979s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345095395 (56h41m38.896565856s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345501417 (56h41m38.896971868s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346057606 (56h41m38.897528067s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346422612 (56h41m38.897893063s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346447920 (56h41m38.897918381s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346801393 (56h41m38.898271854s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347099851 (56h41m38.898570302s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347169985 (56h41m38.898640446s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347569877 (56h41m38.899040328s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347577421 (56h41m38.899047872s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347916837 (56h41m38.899387298s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.348105827 (56h41m38.899576288s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:42.155239479 (56h41m40.70670994s) [upload=49952]: Next returned nil segment Nov 11 16:45:42.155253296 (56h41m40.706723757s) [upload=49952]: Waiting for error group managing segments... Nov 11 16:45:42.509736187 (56h41m41.061206668s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.523781983 (56h41m41.075252444s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.524977681 (56h41m41.076448162s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.544084193 (56h41m41.095554654s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.545880774 (56h41m41.097351236s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.617572702 (56h41m41.169043163s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:49.538637614 (56h41m48.090108075s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.209224227 (56h41m53.760694688s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.376479046 (56h41m53.927949507s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.381095176 (56h41m53.932565637s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:02.516152664 (56h42m1.067623125s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:07.807035175 (56h42m6.358505636s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165478509 (56h42m6.71694897s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165504899 (56h42m6.71697536s) [seg_pos={0 0}, upload=49952]: Segment reached optimal threshold of 13 pieces. Nov 11 16:46:08.165731340 (56h42m6.717201801s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection Nov 11 16:46:08.165742050 (56h42m6.717212521s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.165752790 (56h42m6.717223252s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection Nov 11 16:46:08.165760676 (56h42m6.717231137s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166125290 (56h42m6.717595751s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Nov 11 16:46:08.166135960 (56h42m6.717606421s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166155798 (56h42m6.717626259s) [seg_pos={0 0}, upload=49952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 16:46:08.166239006 (56h42m6.717709467s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 16:46:08.166245079 (56h42m6.71771553s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 16:46:08.467714732 (56h42m7.019185193s) [upload=49952]: Done uploading segments... Nov 11 16:46:08.467737616 (56h42m7.019208077s) [upload=49952]: Upload finished. err: <nil> Duration: 8.533705494s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [........X......] duration: 66.828195ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/21cc4106a0209460 FAIL (2025-11-11 17:04:34 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:51104->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 66.861689ms core website
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 1.432664479s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 17:04:22 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.432695297s core uplink eu1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.........X.....] duration: 65.607441ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/21cc4106a0209460 FAIL (2025-11-11 17:04:07 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:51104->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 65.642279ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [...X...........] duration: 308.069223ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/553c0de2a794f216 FAIL (2025-11-11 17:04:02 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:57188->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 308.111435ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 8.748558763s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 17:03:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 8.74865098s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)