Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.094080681s FAIL (2025-09-23 17:00:33 UTC) orbiter warning: context deadline exceeded Duration: 6.09415361s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.34862276s FAIL (2025-09-23 17:00:01 UTC) orbiter warning: context deadline exceeded Duration: 6.348640553s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.503113728s FAIL (2025-09-23 16:59:11 UTC) orbiter warning: context deadline exceeded Duration: 6.503133686s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.009642072s FAIL (2025-09-23 16:58:01 UTC) orbiter warning: context deadline exceeded Duration: 6.009672591s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.515795285s FAIL (2025-09-23 16:57:07 UTC) orbiter warning: context deadline exceeded Duration: 6.515830884s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.947154926s FAIL (2025-09-23 16:56:41 UTC) orbiter warning: context deadline exceeded Duration: 5.947181176s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.284462127s FAIL (2025-09-23 16:56:13 UTC) orbiter warning: context deadline exceeded Duration: 6.284494288s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.447152217s FAIL (2025-09-23 16:55:41 UTC) orbiter warning: context deadline exceeded Duration: 6.447182494s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.261427095s FAIL (2025-09-23 16:54:35 UTC) orbiter warning: context deadline exceeded Duration: 6.261451132s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.165845894s FAIL (2025-09-23 16:54:00 UTC) orbiter warning: context deadline exceeded Duration: 6.165875s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.983623929s FAIL (2025-09-23 16:53:05 UTC) orbiter warning: context deadline exceeded Duration: 5.98364526s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.178649654s FAIL (2025-09-23 16:52:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA) context deadline exceeded; piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA) context deadline exceeded ======LOGS====== Sep 23 16:08:38.605894181 (2h37m24.067986664s) [upload=2325]: Starting upload Sep 23 16:08:38.606004200 (2h37m24.068096693s) [upload=2325]: Done starting upload Sep 23 16:08:38.606176218 (2h37m24.068268701s) [upload=2325]: Uploading segments... Sep 23 16:08:38.606277210 (2h37m24.068369683s) [upload=2325]: Got next segment. Inline: false Sep 23 16:08:38.606318859 (2h37m24.068411342s) [upload=2325]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 16:08:38.606324821 (2h37m24.068417304s) [upload=2325]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 16:08:38.852132441 (2h37m24.314224925s) [seg_pos={0 0}, upload=2325]: Begin upload segment... Sep 23 16:08:38.852225929 (2h37m24.314318402s) [seg_pos={0 0}, upload=2325]: Done begin upload segment. Sep 23 16:08:38.852268150 (2h37m24.314360623s) [noise=true , address=145.79.227.11:4731 , piece_id=46MQMGAXKX2MFI2IXJL7ZWTBVXYQRM6XA7XD565K2F73AKNSTN6A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.852333363 (2h37m24.314425866s) [noise=true , address=77.90.31.12:4589 , piece_id=IRIEBUXYXWV2WHWNQDZT2ZPE75VHH4W3SURAYJ4MKMZVLIUUAUBQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853073563 (2h37m24.315166037s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853408290 (2h37m24.315500763s) [noise=true , address=45.91.116.11:4117 , piece_id=Q2P4XNRWOQEM257IMW253FIOTAWSPG3IEL55VTB2NHXSAQWAXH2A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853673696 (2h37m24.315766179s) [noise=true , address=45.91.116.5:4298 , piece_id=4T5H6GEBD5CWFHF7WKB4ISNKLF2ZS5DFVRZFWMQR7JUMWD6OFQBA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853967866 (2h37m24.316060349s) [noise=true , address=45.91.116.12:4227 , piece_id=IJTDTA3FDZCCVCCMWN3UWSNTWTSDU6UMY6F24GENRLVD3IXJ33WQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854047137 (2h37m24.31613961s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854304707 (2h37m24.31639718s) [noise=true , address=45.91.116.9:4262 , piece_id=LO2THA7C52KXZGEPEFXDSJOUBQDUIAJCO6UNQF7ZXZ7ETHN4KIOA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854435016 (2h37m24.316527499s) [noise=true , address=77.90.31.10:4519 , piece_id=V7GQEGTIC643FJXEIXRRX64APJQD4YKJFLKEHXFBWXMRUGQ6I4KA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854553952 (2h37m24.316646425s) [noise=true , address=145.79.227.5:4222 , piece_id=HE27KIBGHAZPFT5FUX7FL4KLMWRG3YBZ74N4Q4FGROSK2VFA5S3A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854819237 (2h37m24.31691172s) [noise=true , address=45.91.116.10:4082 , piece_id=YP64RUTIYQWN775JB527KG3KOJQMMIK6AJYRAJLPAIQBQH4NIZVA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854858622 (2h37m24.316951095s) [noise=true , address=45.91.116.12:4225 , piece_id=NF6W3OWZ7CD2XYLUPWAQX6EPGI753JCTD6GYGX6PSIIUQ5OD6SOA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.855251851 (2h37m24.317344324s) [noise=true , address=145.79.227.15:4876 , piece_id=Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.855651973 (2h37m24.317744456s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.856044019 (2h37m24.318136502s) [noise=true , address=145.79.227.19:4079 , piece_id=ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.856461354 (2h37m24.318553827s) [noise=true , address=145.79.227.14:4839 , piece_id=MPMNQZLCBFLMMBPYEQCJRJTTCZGZPZRGPDYW55PBM2I6V5RKXBAQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:40.657981922 (2h37m26.120074406s) [upload=2325]: Next returned nil segment Sep 23 16:08:40.658017650 (2h37m26.120110144s) [upload=2325]: Waiting for error group managing segments... Sep 23 16:08:40.965574730 (2h37m26.427667203s) [noise=true , address=45.91.116.5:4298 , piece_id=4T5H6GEBD5CWFHF7WKB4ISNKLF2ZS5DFVRZFWMQR7JUMWD6OFQBA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.017924353 (2h37m26.480016836s) [noise=true , address=45.91.116.9:4262 , piece_id=LO2THA7C52KXZGEPEFXDSJOUBQDUIAJCO6UNQF7ZXZ7ETHN4KIOA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.055061418 (2h37m26.517153891s) [noise=true , address=45.91.116.12:4225 , piece_id=NF6W3OWZ7CD2XYLUPWAQX6EPGI753JCTD6GYGX6PSIIUQ5OD6SOA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.059138873 (2h37m26.521231346s) [noise=true , address=77.90.31.10:4519 , piece_id=V7GQEGTIC643FJXEIXRRX64APJQD4YKJFLKEHXFBWXMRUGQ6I4KA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.068239702 (2h37m26.530332175s) [noise=true , address=45.91.116.11:4117 , piece_id=Q2P4XNRWOQEM257IMW253FIOTAWSPG3IEL55VTB2NHXSAQWAXH2A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.071035325 (2h37m26.533127798s) [noise=true , address=45.91.116.10:4082 , piece_id=YP64RUTIYQWN775JB527KG3KOJQMMIK6AJYRAJLPAIQBQH4NIZVA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.120390436 (2h37m26.582482919s) [noise=true , address=45.91.116.12:4227 , piece_id=IJTDTA3FDZCCVCCMWN3UWSNTWTSDU6UMY6F24GENRLVD3IXJ33WQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.262045750 (2h37m26.724138234s) [noise=true , address=77.90.31.12:4589 , piece_id=IRIEBUXYXWV2WHWNQDZT2ZPE75VHH4W3SURAYJ4MKMZVLIUUAUBQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:42.626787709 (2h37m28.088880182s) [noise=true , address=145.79.227.19:4079 , piece_id=ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:42.641124409 (2h37m28.103216882s) [noise=true , address=145.79.227.15:4876 , piece_id=Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.056725062 (2h37m28.518817545s) [noise=true , address=145.79.227.14:4839 , piece_id=MPMNQZLCBFLMMBPYEQCJRJTTCZGZPZRGPDYW55PBM2I6V5RKXBAQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.577366321 (2h37m29.039458804s) [noise=true , address=145.79.227.5:4222 , piece_id=HE27KIBGHAZPFT5FUX7FL4KLMWRG3YBZ74N4Q4FGROSK2VFA5S3A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.993428202 (2h37m29.455520675s) [noise=true , address=145.79.227.11:4731 , piece_id=46MQMGAXKX2MFI2IXJL7ZWTBVXYQRM6XA7XD565K2F73AKNSTN6A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.993452629 (2h37m29.455545102s) [seg_pos={0 0}, upload=2325]: Segment reached optimal threshold of 13 pieces. Sep 23 16:08:43.993624226 (2h37m29.455716709s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: expected piece hash; context canceled; EOF Sep 23 16:08:43.993637912 (2h37m29.455730395s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993678629 (2h37m29.455771112s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ): protocol: write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection Sep 23 16:08:43.993693137 (2h37m29.45578561s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993722223 (2h37m29.455814706s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Sep 23 16:08:43.993731370 (2h37m29.455823853s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993751087 (2h37m29.45584356s) [seg_pos={0 0}, upload=2325]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 16:08:43.993849845 (2h37m29.455942328s) [upload=2325]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 16:08:43.993858933 (2h37m29.455951416s) [upload=2325]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 16:08:44.305789411 (2h37m29.767881884s) [upload=2325]: Done uploading segments... Sep 23 16:08:44.305854725 (2h37m29.767947209s) [upload=2325]: Upload finished. err: <nil> Duration: 6.178687716s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.252532829s FAIL (2025-09-23 16:52:12 UTC) orbiter warning: context deadline exceeded Duration: 6.252562706s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.078587548s FAIL (2025-09-23 16:52:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA) context deadline exceeded; piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA) context deadline exceeded ======LOGS====== Sep 23 16:08:38.605894181 (2h37m24.067986664s) [upload=2325]: Starting upload Sep 23 16:08:38.606004200 (2h37m24.068096693s) [upload=2325]: Done starting upload Sep 23 16:08:38.606176218 (2h37m24.068268701s) [upload=2325]: Uploading segments... Sep 23 16:08:38.606277210 (2h37m24.068369683s) [upload=2325]: Got next segment. Inline: false Sep 23 16:08:38.606318859 (2h37m24.068411342s) [upload=2325]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 16:08:38.606324821 (2h37m24.068417304s) [upload=2325]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 16:08:38.852132441 (2h37m24.314224925s) [seg_pos={0 0}, upload=2325]: Begin upload segment... Sep 23 16:08:38.852225929 (2h37m24.314318402s) [seg_pos={0 0}, upload=2325]: Done begin upload segment. Sep 23 16:08:38.852268150 (2h37m24.314360623s) [noise=true , address=145.79.227.11:4731 , piece_id=46MQMGAXKX2MFI2IXJL7ZWTBVXYQRM6XA7XD565K2F73AKNSTN6A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.852333363 (2h37m24.314425866s) [noise=true , address=77.90.31.12:4589 , piece_id=IRIEBUXYXWV2WHWNQDZT2ZPE75VHH4W3SURAYJ4MKMZVLIUUAUBQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853073563 (2h37m24.315166037s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853408290 (2h37m24.315500763s) [noise=true , address=45.91.116.11:4117 , piece_id=Q2P4XNRWOQEM257IMW253FIOTAWSPG3IEL55VTB2NHXSAQWAXH2A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853673696 (2h37m24.315766179s) [noise=true , address=45.91.116.5:4298 , piece_id=4T5H6GEBD5CWFHF7WKB4ISNKLF2ZS5DFVRZFWMQR7JUMWD6OFQBA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853967866 (2h37m24.316060349s) [noise=true , address=45.91.116.12:4227 , piece_id=IJTDTA3FDZCCVCCMWN3UWSNTWTSDU6UMY6F24GENRLVD3IXJ33WQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854047137 (2h37m24.31613961s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854304707 (2h37m24.31639718s) [noise=true , address=45.91.116.9:4262 , piece_id=LO2THA7C52KXZGEPEFXDSJOUBQDUIAJCO6UNQF7ZXZ7ETHN4KIOA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854435016 (2h37m24.316527499s) [noise=true , address=77.90.31.10:4519 , piece_id=V7GQEGTIC643FJXEIXRRX64APJQD4YKJFLKEHXFBWXMRUGQ6I4KA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854553952 (2h37m24.316646425s) [noise=true , address=145.79.227.5:4222 , piece_id=HE27KIBGHAZPFT5FUX7FL4KLMWRG3YBZ74N4Q4FGROSK2VFA5S3A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854819237 (2h37m24.31691172s) [noise=true , address=45.91.116.10:4082 , piece_id=YP64RUTIYQWN775JB527KG3KOJQMMIK6AJYRAJLPAIQBQH4NIZVA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854858622 (2h37m24.316951095s) [noise=true , address=45.91.116.12:4225 , piece_id=NF6W3OWZ7CD2XYLUPWAQX6EPGI753JCTD6GYGX6PSIIUQ5OD6SOA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.855251851 (2h37m24.317344324s) [noise=true , address=145.79.227.15:4876 , piece_id=Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.855651973 (2h37m24.317744456s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.856044019 (2h37m24.318136502s) [noise=true , address=145.79.227.19:4079 , piece_id=ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.856461354 (2h37m24.318553827s) [noise=true , address=145.79.227.14:4839 , piece_id=MPMNQZLCBFLMMBPYEQCJRJTTCZGZPZRGPDYW55PBM2I6V5RKXBAQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:40.657981922 (2h37m26.120074406s) [upload=2325]: Next returned nil segment Sep 23 16:08:40.658017650 (2h37m26.120110144s) [upload=2325]: Waiting for error group managing segments... Sep 23 16:08:40.965574730 (2h37m26.427667203s) [noise=true , address=45.91.116.5:4298 , piece_id=4T5H6GEBD5CWFHF7WKB4ISNKLF2ZS5DFVRZFWMQR7JUMWD6OFQBA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.017924353 (2h37m26.480016836s) [noise=true , address=45.91.116.9:4262 , piece_id=LO2THA7C52KXZGEPEFXDSJOUBQDUIAJCO6UNQF7ZXZ7ETHN4KIOA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.055061418 (2h37m26.517153891s) [noise=true , address=45.91.116.12:4225 , piece_id=NF6W3OWZ7CD2XYLUPWAQX6EPGI753JCTD6GYGX6PSIIUQ5OD6SOA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.059138873 (2h37m26.521231346s) [noise=true , address=77.90.31.10:4519 , piece_id=V7GQEGTIC643FJXEIXRRX64APJQD4YKJFLKEHXFBWXMRUGQ6I4KA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.068239702 (2h37m26.530332175s) [noise=true , address=45.91.116.11:4117 , piece_id=Q2P4XNRWOQEM257IMW253FIOTAWSPG3IEL55VTB2NHXSAQWAXH2A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.071035325 (2h37m26.533127798s) [noise=true , address=45.91.116.10:4082 , piece_id=YP64RUTIYQWN775JB527KG3KOJQMMIK6AJYRAJLPAIQBQH4NIZVA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.120390436 (2h37m26.582482919s) [noise=true , address=45.91.116.12:4227 , piece_id=IJTDTA3FDZCCVCCMWN3UWSNTWTSDU6UMY6F24GENRLVD3IXJ33WQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.262045750 (2h37m26.724138234s) [noise=true , address=77.90.31.12:4589 , piece_id=IRIEBUXYXWV2WHWNQDZT2ZPE75VHH4W3SURAYJ4MKMZVLIUUAUBQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:42.626787709 (2h37m28.088880182s) [noise=true , address=145.79.227.19:4079 , piece_id=ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:42.641124409 (2h37m28.103216882s) [noise=true , address=145.79.227.15:4876 , piece_id=Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.056725062 (2h37m28.518817545s) [noise=true , address=145.79.227.14:4839 , piece_id=MPMNQZLCBFLMMBPYEQCJRJTTCZGZPZRGPDYW55PBM2I6V5RKXBAQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.577366321 (2h37m29.039458804s) [noise=true , address=145.79.227.5:4222 , piece_id=HE27KIBGHAZPFT5FUX7FL4KLMWRG3YBZ74N4Q4FGROSK2VFA5S3A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.993428202 (2h37m29.455520675s) [noise=true , address=145.79.227.11:4731 , piece_id=46MQMGAXKX2MFI2IXJL7ZWTBVXYQRM6XA7XD565K2F73AKNSTN6A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.993452629 (2h37m29.455545102s) [seg_pos={0 0}, upload=2325]: Segment reached optimal threshold of 13 pieces. Sep 23 16:08:43.993624226 (2h37m29.455716709s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: expected piece hash; context canceled; EOF Sep 23 16:08:43.993637912 (2h37m29.455730395s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993678629 (2h37m29.455771112s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ): protocol: write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection Sep 23 16:08:43.993693137 (2h37m29.45578561s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993722223 (2h37m29.455814706s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Sep 23 16:08:43.993731370 (2h37m29.455823853s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993751087 (2h37m29.45584356s) [seg_pos={0 0}, upload=2325]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 16:08:43.993849845 (2h37m29.455942328s) [upload=2325]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 16:08:43.993858933 (2h37m29.455951416s) [upload=2325]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 16:08:44.305789411 (2h37m29.767881884s) [upload=2325]: Done uploading segments... Sep 23 16:08:44.305854725 (2h37m29.767947209s) [upload=2325]: Upload finished. err: <nil> Duration: 6.078624519s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.141154924s FAIL (2025-09-23 16:51:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA) context deadline exceeded; piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA) context deadline exceeded ======LOGS====== Sep 23 16:08:38.605894181 (2h37m24.067986664s) [upload=2325]: Starting upload Sep 23 16:08:38.606004200 (2h37m24.068096693s) [upload=2325]: Done starting upload Sep 23 16:08:38.606176218 (2h37m24.068268701s) [upload=2325]: Uploading segments... Sep 23 16:08:38.606277210 (2h37m24.068369683s) [upload=2325]: Got next segment. Inline: false Sep 23 16:08:38.606318859 (2h37m24.068411342s) [upload=2325]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 16:08:38.606324821 (2h37m24.068417304s) [upload=2325]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 16:08:38.852132441 (2h37m24.314224925s) [seg_pos={0 0}, upload=2325]: Begin upload segment... Sep 23 16:08:38.852225929 (2h37m24.314318402s) [seg_pos={0 0}, upload=2325]: Done begin upload segment. Sep 23 16:08:38.852268150 (2h37m24.314360623s) [noise=true , address=145.79.227.11:4731 , piece_id=46MQMGAXKX2MFI2IXJL7ZWTBVXYQRM6XA7XD565K2F73AKNSTN6A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.852333363 (2h37m24.314425866s) [noise=true , address=77.90.31.12:4589 , piece_id=IRIEBUXYXWV2WHWNQDZT2ZPE75VHH4W3SURAYJ4MKMZVLIUUAUBQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853073563 (2h37m24.315166037s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853408290 (2h37m24.315500763s) [noise=true , address=45.91.116.11:4117 , piece_id=Q2P4XNRWOQEM257IMW253FIOTAWSPG3IEL55VTB2NHXSAQWAXH2A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853673696 (2h37m24.315766179s) [noise=true , address=45.91.116.5:4298 , piece_id=4T5H6GEBD5CWFHF7WKB4ISNKLF2ZS5DFVRZFWMQR7JUMWD6OFQBA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853967866 (2h37m24.316060349s) [noise=true , address=45.91.116.12:4227 , piece_id=IJTDTA3FDZCCVCCMWN3UWSNTWTSDU6UMY6F24GENRLVD3IXJ33WQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854047137 (2h37m24.31613961s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854304707 (2h37m24.31639718s) [noise=true , address=45.91.116.9:4262 , piece_id=LO2THA7C52KXZGEPEFXDSJOUBQDUIAJCO6UNQF7ZXZ7ETHN4KIOA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854435016 (2h37m24.316527499s) [noise=true , address=77.90.31.10:4519 , piece_id=V7GQEGTIC643FJXEIXRRX64APJQD4YKJFLKEHXFBWXMRUGQ6I4KA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854553952 (2h37m24.316646425s) [noise=true , address=145.79.227.5:4222 , piece_id=HE27KIBGHAZPFT5FUX7FL4KLMWRG3YBZ74N4Q4FGROSK2VFA5S3A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854819237 (2h37m24.31691172s) [noise=true , address=45.91.116.10:4082 , piece_id=YP64RUTIYQWN775JB527KG3KOJQMMIK6AJYRAJLPAIQBQH4NIZVA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854858622 (2h37m24.316951095s) [noise=true , address=45.91.116.12:4225 , piece_id=NF6W3OWZ7CD2XYLUPWAQX6EPGI753JCTD6GYGX6PSIIUQ5OD6SOA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.855251851 (2h37m24.317344324s) [noise=true , address=145.79.227.15:4876 , piece_id=Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.855651973 (2h37m24.317744456s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.856044019 (2h37m24.318136502s) [noise=true , address=145.79.227.19:4079 , piece_id=ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.856461354 (2h37m24.318553827s) [noise=true , address=145.79.227.14:4839 , piece_id=MPMNQZLCBFLMMBPYEQCJRJTTCZGZPZRGPDYW55PBM2I6V5RKXBAQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:40.657981922 (2h37m26.120074406s) [upload=2325]: Next returned nil segment Sep 23 16:08:40.658017650 (2h37m26.120110144s) [upload=2325]: Waiting for error group managing segments... Sep 23 16:08:40.965574730 (2h37m26.427667203s) [noise=true , address=45.91.116.5:4298 , piece_id=4T5H6GEBD5CWFHF7WKB4ISNKLF2ZS5DFVRZFWMQR7JUMWD6OFQBA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.017924353 (2h37m26.480016836s) [noise=true , address=45.91.116.9:4262 , piece_id=LO2THA7C52KXZGEPEFXDSJOUBQDUIAJCO6UNQF7ZXZ7ETHN4KIOA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.055061418 (2h37m26.517153891s) [noise=true , address=45.91.116.12:4225 , piece_id=NF6W3OWZ7CD2XYLUPWAQX6EPGI753JCTD6GYGX6PSIIUQ5OD6SOA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.059138873 (2h37m26.521231346s) [noise=true , address=77.90.31.10:4519 , piece_id=V7GQEGTIC643FJXEIXRRX64APJQD4YKJFLKEHXFBWXMRUGQ6I4KA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.068239702 (2h37m26.530332175s) [noise=true , address=45.91.116.11:4117 , piece_id=Q2P4XNRWOQEM257IMW253FIOTAWSPG3IEL55VTB2NHXSAQWAXH2A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.071035325 (2h37m26.533127798s) [noise=true , address=45.91.116.10:4082 , piece_id=YP64RUTIYQWN775JB527KG3KOJQMMIK6AJYRAJLPAIQBQH4NIZVA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.120390436 (2h37m26.582482919s) [noise=true , address=45.91.116.12:4227 , piece_id=IJTDTA3FDZCCVCCMWN3UWSNTWTSDU6UMY6F24GENRLVD3IXJ33WQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.262045750 (2h37m26.724138234s) [noise=true , address=77.90.31.12:4589 , piece_id=IRIEBUXYXWV2WHWNQDZT2ZPE75VHH4W3SURAYJ4MKMZVLIUUAUBQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:42.626787709 (2h37m28.088880182s) [noise=true , address=145.79.227.19:4079 , piece_id=ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:42.641124409 (2h37m28.103216882s) [noise=true , address=145.79.227.15:4876 , piece_id=Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.056725062 (2h37m28.518817545s) [noise=true , address=145.79.227.14:4839 , piece_id=MPMNQZLCBFLMMBPYEQCJRJTTCZGZPZRGPDYW55PBM2I6V5RKXBAQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.577366321 (2h37m29.039458804s) [noise=true , address=145.79.227.5:4222 , piece_id=HE27KIBGHAZPFT5FUX7FL4KLMWRG3YBZ74N4Q4FGROSK2VFA5S3A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.993428202 (2h37m29.455520675s) [noise=true , address=145.79.227.11:4731 , piece_id=46MQMGAXKX2MFI2IXJL7ZWTBVXYQRM6XA7XD565K2F73AKNSTN6A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.993452629 (2h37m29.455545102s) [seg_pos={0 0}, upload=2325]: Segment reached optimal threshold of 13 pieces. Sep 23 16:08:43.993624226 (2h37m29.455716709s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: expected piece hash; context canceled; EOF Sep 23 16:08:43.993637912 (2h37m29.455730395s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993678629 (2h37m29.455771112s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ): protocol: write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection Sep 23 16:08:43.993693137 (2h37m29.45578561s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993722223 (2h37m29.455814706s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Sep 23 16:08:43.993731370 (2h37m29.455823853s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993751087 (2h37m29.45584356s) [seg_pos={0 0}, upload=2325]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 16:08:43.993849845 (2h37m29.455942328s) [upload=2325]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 16:08:43.993858933 (2h37m29.455951416s) [upload=2325]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 16:08:44.305789411 (2h37m29.767881884s) [upload=2325]: Done uploading segments... Sep 23 16:08:44.305854725 (2h37m29.767947209s) [upload=2325]: Upload finished. err: <nil> Duration: 6.141178098s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.098274022s FAIL (2025-09-23 16:51:16 UTC) orbiter warning: context deadline exceeded Duration: 6.098296847s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.395386437s FAIL (2025-09-23 16:50:35 UTC) orbiter warning: context deadline exceeded Duration: 6.395414069s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.144843047s FAIL (2025-09-23 16:50:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA) context deadline exceeded; piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA) context deadline exceeded ======LOGS====== Sep 23 16:08:38.605894181 (2h37m24.067986664s) [upload=2325]: Starting upload Sep 23 16:08:38.606004200 (2h37m24.068096693s) [upload=2325]: Done starting upload Sep 23 16:08:38.606176218 (2h37m24.068268701s) [upload=2325]: Uploading segments... Sep 23 16:08:38.606277210 (2h37m24.068369683s) [upload=2325]: Got next segment. Inline: false Sep 23 16:08:38.606318859 (2h37m24.068411342s) [upload=2325]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 16:08:38.606324821 (2h37m24.068417304s) [upload=2325]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 16:08:38.852132441 (2h37m24.314224925s) [seg_pos={0 0}, upload=2325]: Begin upload segment... Sep 23 16:08:38.852225929 (2h37m24.314318402s) [seg_pos={0 0}, upload=2325]: Done begin upload segment. Sep 23 16:08:38.852268150 (2h37m24.314360623s) [noise=true , address=145.79.227.11:4731 , piece_id=46MQMGAXKX2MFI2IXJL7ZWTBVXYQRM6XA7XD565K2F73AKNSTN6A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.852333363 (2h37m24.314425866s) [noise=true , address=77.90.31.12:4589 , piece_id=IRIEBUXYXWV2WHWNQDZT2ZPE75VHH4W3SURAYJ4MKMZVLIUUAUBQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853073563 (2h37m24.315166037s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853408290 (2h37m24.315500763s) [noise=true , address=45.91.116.11:4117 , piece_id=Q2P4XNRWOQEM257IMW253FIOTAWSPG3IEL55VTB2NHXSAQWAXH2A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853673696 (2h37m24.315766179s) [noise=true , address=45.91.116.5:4298 , piece_id=4T5H6GEBD5CWFHF7WKB4ISNKLF2ZS5DFVRZFWMQR7JUMWD6OFQBA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853967866 (2h37m24.316060349s) [noise=true , address=45.91.116.12:4227 , piece_id=IJTDTA3FDZCCVCCMWN3UWSNTWTSDU6UMY6F24GENRLVD3IXJ33WQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854047137 (2h37m24.31613961s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854304707 (2h37m24.31639718s) [noise=true , address=45.91.116.9:4262 , piece_id=LO2THA7C52KXZGEPEFXDSJOUBQDUIAJCO6UNQF7ZXZ7ETHN4KIOA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854435016 (2h37m24.316527499s) [noise=true , address=77.90.31.10:4519 , piece_id=V7GQEGTIC643FJXEIXRRX64APJQD4YKJFLKEHXFBWXMRUGQ6I4KA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854553952 (2h37m24.316646425s) [noise=true , address=145.79.227.5:4222 , piece_id=HE27KIBGHAZPFT5FUX7FL4KLMWRG3YBZ74N4Q4FGROSK2VFA5S3A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854819237 (2h37m24.31691172s) [noise=true , address=45.91.116.10:4082 , piece_id=YP64RUTIYQWN775JB527KG3KOJQMMIK6AJYRAJLPAIQBQH4NIZVA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854858622 (2h37m24.316951095s) [noise=true , address=45.91.116.12:4225 , piece_id=NF6W3OWZ7CD2XYLUPWAQX6EPGI753JCTD6GYGX6PSIIUQ5OD6SOA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.855251851 (2h37m24.317344324s) [noise=true , address=145.79.227.15:4876 , piece_id=Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.855651973 (2h37m24.317744456s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.856044019 (2h37m24.318136502s) [noise=true , address=145.79.227.19:4079 , piece_id=ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.856461354 (2h37m24.318553827s) [noise=true , address=145.79.227.14:4839 , piece_id=MPMNQZLCBFLMMBPYEQCJRJTTCZGZPZRGPDYW55PBM2I6V5RKXBAQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:40.657981922 (2h37m26.120074406s) [upload=2325]: Next returned nil segment Sep 23 16:08:40.658017650 (2h37m26.120110144s) [upload=2325]: Waiting for error group managing segments... Sep 23 16:08:40.965574730 (2h37m26.427667203s) [noise=true , address=45.91.116.5:4298 , piece_id=4T5H6GEBD5CWFHF7WKB4ISNKLF2ZS5DFVRZFWMQR7JUMWD6OFQBA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.017924353 (2h37m26.480016836s) [noise=true , address=45.91.116.9:4262 , piece_id=LO2THA7C52KXZGEPEFXDSJOUBQDUIAJCO6UNQF7ZXZ7ETHN4KIOA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.055061418 (2h37m26.517153891s) [noise=true , address=45.91.116.12:4225 , piece_id=NF6W3OWZ7CD2XYLUPWAQX6EPGI753JCTD6GYGX6PSIIUQ5OD6SOA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.059138873 (2h37m26.521231346s) [noise=true , address=77.90.31.10:4519 , piece_id=V7GQEGTIC643FJXEIXRRX64APJQD4YKJFLKEHXFBWXMRUGQ6I4KA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.068239702 (2h37m26.530332175s) [noise=true , address=45.91.116.11:4117 , piece_id=Q2P4XNRWOQEM257IMW253FIOTAWSPG3IEL55VTB2NHXSAQWAXH2A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.071035325 (2h37m26.533127798s) [noise=true , address=45.91.116.10:4082 , piece_id=YP64RUTIYQWN775JB527KG3KOJQMMIK6AJYRAJLPAIQBQH4NIZVA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.120390436 (2h37m26.582482919s) [noise=true , address=45.91.116.12:4227 , piece_id=IJTDTA3FDZCCVCCMWN3UWSNTWTSDU6UMY6F24GENRLVD3IXJ33WQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.262045750 (2h37m26.724138234s) [noise=true , address=77.90.31.12:4589 , piece_id=IRIEBUXYXWV2WHWNQDZT2ZPE75VHH4W3SURAYJ4MKMZVLIUUAUBQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:42.626787709 (2h37m28.088880182s) [noise=true , address=145.79.227.19:4079 , piece_id=ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:42.641124409 (2h37m28.103216882s) [noise=true , address=145.79.227.15:4876 , piece_id=Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.056725062 (2h37m28.518817545s) [noise=true , address=145.79.227.14:4839 , piece_id=MPMNQZLCBFLMMBPYEQCJRJTTCZGZPZRGPDYW55PBM2I6V5RKXBAQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.577366321 (2h37m29.039458804s) [noise=true , address=145.79.227.5:4222 , piece_id=HE27KIBGHAZPFT5FUX7FL4KLMWRG3YBZ74N4Q4FGROSK2VFA5S3A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.993428202 (2h37m29.455520675s) [noise=true , address=145.79.227.11:4731 , piece_id=46MQMGAXKX2MFI2IXJL7ZWTBVXYQRM6XA7XD565K2F73AKNSTN6A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.993452629 (2h37m29.455545102s) [seg_pos={0 0}, upload=2325]: Segment reached optimal threshold of 13 pieces. Sep 23 16:08:43.993624226 (2h37m29.455716709s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: expected piece hash; context canceled; EOF Sep 23 16:08:43.993637912 (2h37m29.455730395s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993678629 (2h37m29.455771112s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ): protocol: write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection Sep 23 16:08:43.993693137 (2h37m29.45578561s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993722223 (2h37m29.455814706s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Sep 23 16:08:43.993731370 (2h37m29.455823853s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993751087 (2h37m29.45584356s) [seg_pos={0 0}, upload=2325]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 16:08:43.993849845 (2h37m29.455942328s) [upload=2325]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 16:08:43.993858933 (2h37m29.455951416s) [upload=2325]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 16:08:44.305789411 (2h37m29.767881884s) [upload=2325]: Done uploading segments... Sep 23 16:08:44.305854725 (2h37m29.767947209s) [upload=2325]: Upload finished. err: <nil> Duration: 6.144869268s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.982756978s FAIL (2025-09-23 16:50:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA) context deadline exceeded; piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA) context deadline exceeded ======LOGS====== Sep 23 16:08:38.605894181 (2h37m24.067986664s) [upload=2325]: Starting upload Sep 23 16:08:38.606004200 (2h37m24.068096693s) [upload=2325]: Done starting upload Sep 23 16:08:38.606176218 (2h37m24.068268701s) [upload=2325]: Uploading segments... Sep 23 16:08:38.606277210 (2h37m24.068369683s) [upload=2325]: Got next segment. Inline: false Sep 23 16:08:38.606318859 (2h37m24.068411342s) [upload=2325]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 16:08:38.606324821 (2h37m24.068417304s) [upload=2325]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 16:08:38.852132441 (2h37m24.314224925s) [seg_pos={0 0}, upload=2325]: Begin upload segment... Sep 23 16:08:38.852225929 (2h37m24.314318402s) [seg_pos={0 0}, upload=2325]: Done begin upload segment. Sep 23 16:08:38.852268150 (2h37m24.314360623s) [noise=true , address=145.79.227.11:4731 , piece_id=46MQMGAXKX2MFI2IXJL7ZWTBVXYQRM6XA7XD565K2F73AKNSTN6A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.852333363 (2h37m24.314425866s) [noise=true , address=77.90.31.12:4589 , piece_id=IRIEBUXYXWV2WHWNQDZT2ZPE75VHH4W3SURAYJ4MKMZVLIUUAUBQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853073563 (2h37m24.315166037s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853408290 (2h37m24.315500763s) [noise=true , address=45.91.116.11:4117 , piece_id=Q2P4XNRWOQEM257IMW253FIOTAWSPG3IEL55VTB2NHXSAQWAXH2A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853673696 (2h37m24.315766179s) [noise=true , address=45.91.116.5:4298 , piece_id=4T5H6GEBD5CWFHF7WKB4ISNKLF2ZS5DFVRZFWMQR7JUMWD6OFQBA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853967866 (2h37m24.316060349s) [noise=true , address=45.91.116.12:4227 , piece_id=IJTDTA3FDZCCVCCMWN3UWSNTWTSDU6UMY6F24GENRLVD3IXJ33WQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854047137 (2h37m24.31613961s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854304707 (2h37m24.31639718s) [noise=true , address=45.91.116.9:4262 , piece_id=LO2THA7C52KXZGEPEFXDSJOUBQDUIAJCO6UNQF7ZXZ7ETHN4KIOA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854435016 (2h37m24.316527499s) [noise=true , address=77.90.31.10:4519 , piece_id=V7GQEGTIC643FJXEIXRRX64APJQD4YKJFLKEHXFBWXMRUGQ6I4KA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854553952 (2h37m24.316646425s) [noise=true , address=145.79.227.5:4222 , piece_id=HE27KIBGHAZPFT5FUX7FL4KLMWRG3YBZ74N4Q4FGROSK2VFA5S3A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854819237 (2h37m24.31691172s) [noise=true , address=45.91.116.10:4082 , piece_id=YP64RUTIYQWN775JB527KG3KOJQMMIK6AJYRAJLPAIQBQH4NIZVA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854858622 (2h37m24.316951095s) [noise=true , address=45.91.116.12:4225 , piece_id=NF6W3OWZ7CD2XYLUPWAQX6EPGI753JCTD6GYGX6PSIIUQ5OD6SOA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.855251851 (2h37m24.317344324s) [noise=true , address=145.79.227.15:4876 , piece_id=Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.855651973 (2h37m24.317744456s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.856044019 (2h37m24.318136502s) [noise=true , address=145.79.227.19:4079 , piece_id=ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.856461354 (2h37m24.318553827s) [noise=true , address=145.79.227.14:4839 , piece_id=MPMNQZLCBFLMMBPYEQCJRJTTCZGZPZRGPDYW55PBM2I6V5RKXBAQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:40.657981922 (2h37m26.120074406s) [upload=2325]: Next returned nil segment Sep 23 16:08:40.658017650 (2h37m26.120110144s) [upload=2325]: Waiting for error group managing segments... Sep 23 16:08:40.965574730 (2h37m26.427667203s) [noise=true , address=45.91.116.5:4298 , piece_id=4T5H6GEBD5CWFHF7WKB4ISNKLF2ZS5DFVRZFWMQR7JUMWD6OFQBA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.017924353 (2h37m26.480016836s) [noise=true , address=45.91.116.9:4262 , piece_id=LO2THA7C52KXZGEPEFXDSJOUBQDUIAJCO6UNQF7ZXZ7ETHN4KIOA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.055061418 (2h37m26.517153891s) [noise=true , address=45.91.116.12:4225 , piece_id=NF6W3OWZ7CD2XYLUPWAQX6EPGI753JCTD6GYGX6PSIIUQ5OD6SOA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.059138873 (2h37m26.521231346s) [noise=true , address=77.90.31.10:4519 , piece_id=V7GQEGTIC643FJXEIXRRX64APJQD4YKJFLKEHXFBWXMRUGQ6I4KA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.068239702 (2h37m26.530332175s) [noise=true , address=45.91.116.11:4117 , piece_id=Q2P4XNRWOQEM257IMW253FIOTAWSPG3IEL55VTB2NHXSAQWAXH2A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.071035325 (2h37m26.533127798s) [noise=true , address=45.91.116.10:4082 , piece_id=YP64RUTIYQWN775JB527KG3KOJQMMIK6AJYRAJLPAIQBQH4NIZVA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.120390436 (2h37m26.582482919s) [noise=true , address=45.91.116.12:4227 , piece_id=IJTDTA3FDZCCVCCMWN3UWSNTWTSDU6UMY6F24GENRLVD3IXJ33WQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.262045750 (2h37m26.724138234s) [noise=true , address=77.90.31.12:4589 , piece_id=IRIEBUXYXWV2WHWNQDZT2ZPE75VHH4W3SURAYJ4MKMZVLIUUAUBQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:42.626787709 (2h37m28.088880182s) [noise=true , address=145.79.227.19:4079 , piece_id=ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:42.641124409 (2h37m28.103216882s) [noise=true , address=145.79.227.15:4876 , piece_id=Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.056725062 (2h37m28.518817545s) [noise=true , address=145.79.227.14:4839 , piece_id=MPMNQZLCBFLMMBPYEQCJRJTTCZGZPZRGPDYW55PBM2I6V5RKXBAQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.577366321 (2h37m29.039458804s) [noise=true , address=145.79.227.5:4222 , piece_id=HE27KIBGHAZPFT5FUX7FL4KLMWRG3YBZ74N4Q4FGROSK2VFA5S3A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.993428202 (2h37m29.455520675s) [noise=true , address=145.79.227.11:4731 , piece_id=46MQMGAXKX2MFI2IXJL7ZWTBVXYQRM6XA7XD565K2F73AKNSTN6A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.993452629 (2h37m29.455545102s) [seg_pos={0 0}, upload=2325]: Segment reached optimal threshold of 13 pieces. Sep 23 16:08:43.993624226 (2h37m29.455716709s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: expected piece hash; context canceled; EOF Sep 23 16:08:43.993637912 (2h37m29.455730395s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993678629 (2h37m29.455771112s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ): protocol: write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection Sep 23 16:08:43.993693137 (2h37m29.45578561s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993722223 (2h37m29.455814706s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Sep 23 16:08:43.993731370 (2h37m29.455823853s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993751087 (2h37m29.45584356s) [seg_pos={0 0}, upload=2325]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 16:08:43.993849845 (2h37m29.455942328s) [upload=2325]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 16:08:43.993858933 (2h37m29.455951416s) [upload=2325]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 16:08:44.305789411 (2h37m29.767881884s) [upload=2325]: Done uploading segments... Sep 23 16:08:44.305854725 (2h37m29.767947209s) [upload=2325]: Upload finished. err: <nil> Duration: 5.982788699s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.199446294s FAIL (2025-09-23 16:49:30 UTC) orbiter warning: context deadline exceeded Duration: 6.199484217s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.219986465s FAIL (2025-09-23 16:48:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA) context deadline exceeded; piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA) context deadline exceeded ======LOGS====== Sep 23 16:08:38.605894181 (2h37m24.067986664s) [upload=2325]: Starting upload Sep 23 16:08:38.606004200 (2h37m24.068096693s) [upload=2325]: Done starting upload Sep 23 16:08:38.606176218 (2h37m24.068268701s) [upload=2325]: Uploading segments... Sep 23 16:08:38.606277210 (2h37m24.068369683s) [upload=2325]: Got next segment. Inline: false Sep 23 16:08:38.606318859 (2h37m24.068411342s) [upload=2325]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 16:08:38.606324821 (2h37m24.068417304s) [upload=2325]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 16:08:38.852132441 (2h37m24.314224925s) [seg_pos={0 0}, upload=2325]: Begin upload segment... Sep 23 16:08:38.852225929 (2h37m24.314318402s) [seg_pos={0 0}, upload=2325]: Done begin upload segment. Sep 23 16:08:38.852268150 (2h37m24.314360623s) [noise=true , address=145.79.227.11:4731 , piece_id=46MQMGAXKX2MFI2IXJL7ZWTBVXYQRM6XA7XD565K2F73AKNSTN6A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.852333363 (2h37m24.314425866s) [noise=true , address=77.90.31.12:4589 , piece_id=IRIEBUXYXWV2WHWNQDZT2ZPE75VHH4W3SURAYJ4MKMZVLIUUAUBQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853073563 (2h37m24.315166037s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853408290 (2h37m24.315500763s) [noise=true , address=45.91.116.11:4117 , piece_id=Q2P4XNRWOQEM257IMW253FIOTAWSPG3IEL55VTB2NHXSAQWAXH2A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853673696 (2h37m24.315766179s) [noise=true , address=45.91.116.5:4298 , piece_id=4T5H6GEBD5CWFHF7WKB4ISNKLF2ZS5DFVRZFWMQR7JUMWD6OFQBA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853967866 (2h37m24.316060349s) [noise=true , address=45.91.116.12:4227 , piece_id=IJTDTA3FDZCCVCCMWN3UWSNTWTSDU6UMY6F24GENRLVD3IXJ33WQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854047137 (2h37m24.31613961s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854304707 (2h37m24.31639718s) [noise=true , address=45.91.116.9:4262 , piece_id=LO2THA7C52KXZGEPEFXDSJOUBQDUIAJCO6UNQF7ZXZ7ETHN4KIOA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854435016 (2h37m24.316527499s) [noise=true , address=77.90.31.10:4519 , piece_id=V7GQEGTIC643FJXEIXRRX64APJQD4YKJFLKEHXFBWXMRUGQ6I4KA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854553952 (2h37m24.316646425s) [noise=true , address=145.79.227.5:4222 , piece_id=HE27KIBGHAZPFT5FUX7FL4KLMWRG3YBZ74N4Q4FGROSK2VFA5S3A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854819237 (2h37m24.31691172s) [noise=true , address=45.91.116.10:4082 , piece_id=YP64RUTIYQWN775JB527KG3KOJQMMIK6AJYRAJLPAIQBQH4NIZVA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854858622 (2h37m24.316951095s) [noise=true , address=45.91.116.12:4225 , piece_id=NF6W3OWZ7CD2XYLUPWAQX6EPGI753JCTD6GYGX6PSIIUQ5OD6SOA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.855251851 (2h37m24.317344324s) [noise=true , address=145.79.227.15:4876 , piece_id=Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.855651973 (2h37m24.317744456s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.856044019 (2h37m24.318136502s) [noise=true , address=145.79.227.19:4079 , piece_id=ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.856461354 (2h37m24.318553827s) [noise=true , address=145.79.227.14:4839 , piece_id=MPMNQZLCBFLMMBPYEQCJRJTTCZGZPZRGPDYW55PBM2I6V5RKXBAQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:40.657981922 (2h37m26.120074406s) [upload=2325]: Next returned nil segment Sep 23 16:08:40.658017650 (2h37m26.120110144s) [upload=2325]: Waiting for error group managing segments... Sep 23 16:08:40.965574730 (2h37m26.427667203s) [noise=true , address=45.91.116.5:4298 , piece_id=4T5H6GEBD5CWFHF7WKB4ISNKLF2ZS5DFVRZFWMQR7JUMWD6OFQBA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.017924353 (2h37m26.480016836s) [noise=true , address=45.91.116.9:4262 , piece_id=LO2THA7C52KXZGEPEFXDSJOUBQDUIAJCO6UNQF7ZXZ7ETHN4KIOA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.055061418 (2h37m26.517153891s) [noise=true , address=45.91.116.12:4225 , piece_id=NF6W3OWZ7CD2XYLUPWAQX6EPGI753JCTD6GYGX6PSIIUQ5OD6SOA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.059138873 (2h37m26.521231346s) [noise=true , address=77.90.31.10:4519 , piece_id=V7GQEGTIC643FJXEIXRRX64APJQD4YKJFLKEHXFBWXMRUGQ6I4KA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.068239702 (2h37m26.530332175s) [noise=true , address=45.91.116.11:4117 , piece_id=Q2P4XNRWOQEM257IMW253FIOTAWSPG3IEL55VTB2NHXSAQWAXH2A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.071035325 (2h37m26.533127798s) [noise=true , address=45.91.116.10:4082 , piece_id=YP64RUTIYQWN775JB527KG3KOJQMMIK6AJYRAJLPAIQBQH4NIZVA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.120390436 (2h37m26.582482919s) [noise=true , address=45.91.116.12:4227 , piece_id=IJTDTA3FDZCCVCCMWN3UWSNTWTSDU6UMY6F24GENRLVD3IXJ33WQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.262045750 (2h37m26.724138234s) [noise=true , address=77.90.31.12:4589 , piece_id=IRIEBUXYXWV2WHWNQDZT2ZPE75VHH4W3SURAYJ4MKMZVLIUUAUBQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:42.626787709 (2h37m28.088880182s) [noise=true , address=145.79.227.19:4079 , piece_id=ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:42.641124409 (2h37m28.103216882s) [noise=true , address=145.79.227.15:4876 , piece_id=Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.056725062 (2h37m28.518817545s) [noise=true , address=145.79.227.14:4839 , piece_id=MPMNQZLCBFLMMBPYEQCJRJTTCZGZPZRGPDYW55PBM2I6V5RKXBAQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.577366321 (2h37m29.039458804s) [noise=true , address=145.79.227.5:4222 , piece_id=HE27KIBGHAZPFT5FUX7FL4KLMWRG3YBZ74N4Q4FGROSK2VFA5S3A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.993428202 (2h37m29.455520675s) [noise=true , address=145.79.227.11:4731 , piece_id=46MQMGAXKX2MFI2IXJL7ZWTBVXYQRM6XA7XD565K2F73AKNSTN6A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.993452629 (2h37m29.455545102s) [seg_pos={0 0}, upload=2325]: Segment reached optimal threshold of 13 pieces. Sep 23 16:08:43.993624226 (2h37m29.455716709s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: expected piece hash; context canceled; EOF Sep 23 16:08:43.993637912 (2h37m29.455730395s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993678629 (2h37m29.455771112s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ): protocol: write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection Sep 23 16:08:43.993693137 (2h37m29.45578561s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993722223 (2h37m29.455814706s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Sep 23 16:08:43.993731370 (2h37m29.455823853s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993751087 (2h37m29.45584356s) [seg_pos={0 0}, upload=2325]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 16:08:43.993849845 (2h37m29.455942328s) [upload=2325]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 16:08:43.993858933 (2h37m29.455951416s) [upload=2325]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 16:08:44.305789411 (2h37m29.767881884s) [upload=2325]: Done uploading segments... Sep 23 16:08:44.305854725 (2h37m29.767947209s) [upload=2325]: Upload finished. err: <nil> Duration: 6.220014168s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.02814766s FAIL (2025-09-23 16:48:45 UTC) orbiter warning: context deadline exceeded Duration: 6.028184941s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.086901447s FAIL (2025-09-23 16:48:18 UTC) orbiter warning: context deadline exceeded Duration: 6.086924202s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.214408803s FAIL (2025-09-23 16:48:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA) context deadline exceeded; piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA) context deadline exceeded ======LOGS====== Sep 23 16:08:38.605894181 (2h37m24.067986664s) [upload=2325]: Starting upload Sep 23 16:08:38.606004200 (2h37m24.068096693s) [upload=2325]: Done starting upload Sep 23 16:08:38.606176218 (2h37m24.068268701s) [upload=2325]: Uploading segments... Sep 23 16:08:38.606277210 (2h37m24.068369683s) [upload=2325]: Got next segment. Inline: false Sep 23 16:08:38.606318859 (2h37m24.068411342s) [upload=2325]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 16:08:38.606324821 (2h37m24.068417304s) [upload=2325]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 16:08:38.852132441 (2h37m24.314224925s) [seg_pos={0 0}, upload=2325]: Begin upload segment... Sep 23 16:08:38.852225929 (2h37m24.314318402s) [seg_pos={0 0}, upload=2325]: Done begin upload segment. Sep 23 16:08:38.852268150 (2h37m24.314360623s) [noise=true , address=145.79.227.11:4731 , piece_id=46MQMGAXKX2MFI2IXJL7ZWTBVXYQRM6XA7XD565K2F73AKNSTN6A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.852333363 (2h37m24.314425866s) [noise=true , address=77.90.31.12:4589 , piece_id=IRIEBUXYXWV2WHWNQDZT2ZPE75VHH4W3SURAYJ4MKMZVLIUUAUBQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853073563 (2h37m24.315166037s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853408290 (2h37m24.315500763s) [noise=true , address=45.91.116.11:4117 , piece_id=Q2P4XNRWOQEM257IMW253FIOTAWSPG3IEL55VTB2NHXSAQWAXH2A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853673696 (2h37m24.315766179s) [noise=true , address=45.91.116.5:4298 , piece_id=4T5H6GEBD5CWFHF7WKB4ISNKLF2ZS5DFVRZFWMQR7JUMWD6OFQBA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853967866 (2h37m24.316060349s) [noise=true , address=45.91.116.12:4227 , piece_id=IJTDTA3FDZCCVCCMWN3UWSNTWTSDU6UMY6F24GENRLVD3IXJ33WQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854047137 (2h37m24.31613961s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854304707 (2h37m24.31639718s) [noise=true , address=45.91.116.9:4262 , piece_id=LO2THA7C52KXZGEPEFXDSJOUBQDUIAJCO6UNQF7ZXZ7ETHN4KIOA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854435016 (2h37m24.316527499s) [noise=true , address=77.90.31.10:4519 , piece_id=V7GQEGTIC643FJXEIXRRX64APJQD4YKJFLKEHXFBWXMRUGQ6I4KA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854553952 (2h37m24.316646425s) [noise=true , address=145.79.227.5:4222 , piece_id=HE27KIBGHAZPFT5FUX7FL4KLMWRG3YBZ74N4Q4FGROSK2VFA5S3A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854819237 (2h37m24.31691172s) [noise=true , address=45.91.116.10:4082 , piece_id=YP64RUTIYQWN775JB527KG3KOJQMMIK6AJYRAJLPAIQBQH4NIZVA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854858622 (2h37m24.316951095s) [noise=true , address=45.91.116.12:4225 , piece_id=NF6W3OWZ7CD2XYLUPWAQX6EPGI753JCTD6GYGX6PSIIUQ5OD6SOA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.855251851 (2h37m24.317344324s) [noise=true , address=145.79.227.15:4876 , piece_id=Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.855651973 (2h37m24.317744456s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.856044019 (2h37m24.318136502s) [noise=true , address=145.79.227.19:4079 , piece_id=ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.856461354 (2h37m24.318553827s) [noise=true , address=145.79.227.14:4839 , piece_id=MPMNQZLCBFLMMBPYEQCJRJTTCZGZPZRGPDYW55PBM2I6V5RKXBAQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:40.657981922 (2h37m26.120074406s) [upload=2325]: Next returned nil segment Sep 23 16:08:40.658017650 (2h37m26.120110144s) [upload=2325]: Waiting for error group managing segments... Sep 23 16:08:40.965574730 (2h37m26.427667203s) [noise=true , address=45.91.116.5:4298 , piece_id=4T5H6GEBD5CWFHF7WKB4ISNKLF2ZS5DFVRZFWMQR7JUMWD6OFQBA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.017924353 (2h37m26.480016836s) [noise=true , address=45.91.116.9:4262 , piece_id=LO2THA7C52KXZGEPEFXDSJOUBQDUIAJCO6UNQF7ZXZ7ETHN4KIOA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.055061418 (2h37m26.517153891s) [noise=true , address=45.91.116.12:4225 , piece_id=NF6W3OWZ7CD2XYLUPWAQX6EPGI753JCTD6GYGX6PSIIUQ5OD6SOA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.059138873 (2h37m26.521231346s) [noise=true , address=77.90.31.10:4519 , piece_id=V7GQEGTIC643FJXEIXRRX64APJQD4YKJFLKEHXFBWXMRUGQ6I4KA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.068239702 (2h37m26.530332175s) [noise=true , address=45.91.116.11:4117 , piece_id=Q2P4XNRWOQEM257IMW253FIOTAWSPG3IEL55VTB2NHXSAQWAXH2A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.071035325 (2h37m26.533127798s) [noise=true , address=45.91.116.10:4082 , piece_id=YP64RUTIYQWN775JB527KG3KOJQMMIK6AJYRAJLPAIQBQH4NIZVA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.120390436 (2h37m26.582482919s) [noise=true , address=45.91.116.12:4227 , piece_id=IJTDTA3FDZCCVCCMWN3UWSNTWTSDU6UMY6F24GENRLVD3IXJ33WQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.262045750 (2h37m26.724138234s) [noise=true , address=77.90.31.12:4589 , piece_id=IRIEBUXYXWV2WHWNQDZT2ZPE75VHH4W3SURAYJ4MKMZVLIUUAUBQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:42.626787709 (2h37m28.088880182s) [noise=true , address=145.79.227.19:4079 , piece_id=ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:42.641124409 (2h37m28.103216882s) [noise=true , address=145.79.227.15:4876 , piece_id=Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.056725062 (2h37m28.518817545s) [noise=true , address=145.79.227.14:4839 , piece_id=MPMNQZLCBFLMMBPYEQCJRJTTCZGZPZRGPDYW55PBM2I6V5RKXBAQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.577366321 (2h37m29.039458804s) [noise=true , address=145.79.227.5:4222 , piece_id=HE27KIBGHAZPFT5FUX7FL4KLMWRG3YBZ74N4Q4FGROSK2VFA5S3A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.993428202 (2h37m29.455520675s) [noise=true , address=145.79.227.11:4731 , piece_id=46MQMGAXKX2MFI2IXJL7ZWTBVXYQRM6XA7XD565K2F73AKNSTN6A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.993452629 (2h37m29.455545102s) [seg_pos={0 0}, upload=2325]: Segment reached optimal threshold of 13 pieces. Sep 23 16:08:43.993624226 (2h37m29.455716709s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: expected piece hash; context canceled; EOF Sep 23 16:08:43.993637912 (2h37m29.455730395s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993678629 (2h37m29.455771112s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ): protocol: write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection Sep 23 16:08:43.993693137 (2h37m29.45578561s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993722223 (2h37m29.455814706s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Sep 23 16:08:43.993731370 (2h37m29.455823853s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993751087 (2h37m29.45584356s) [seg_pos={0 0}, upload=2325]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 16:08:43.993849845 (2h37m29.455942328s) [upload=2325]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 16:08:43.993858933 (2h37m29.455951416s) [upload=2325]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 16:08:44.305789411 (2h37m29.767881884s) [upload=2325]: Done uploading segments... Sep 23 16:08:44.305854725 (2h37m29.767947209s) [upload=2325]: Upload finished. err: <nil> Duration: 6.214488185s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.034387733s FAIL (2025-09-23 16:47:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA) context deadline exceeded; piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA) context deadline exceeded ======LOGS====== Sep 23 16:08:38.605894181 (2h37m24.067986664s) [upload=2325]: Starting upload Sep 23 16:08:38.606004200 (2h37m24.068096693s) [upload=2325]: Done starting upload Sep 23 16:08:38.606176218 (2h37m24.068268701s) [upload=2325]: Uploading segments... Sep 23 16:08:38.606277210 (2h37m24.068369683s) [upload=2325]: Got next segment. Inline: false Sep 23 16:08:38.606318859 (2h37m24.068411342s) [upload=2325]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 16:08:38.606324821 (2h37m24.068417304s) [upload=2325]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 16:08:38.852132441 (2h37m24.314224925s) [seg_pos={0 0}, upload=2325]: Begin upload segment... Sep 23 16:08:38.852225929 (2h37m24.314318402s) [seg_pos={0 0}, upload=2325]: Done begin upload segment. Sep 23 16:08:38.852268150 (2h37m24.314360623s) [noise=true , address=145.79.227.11:4731 , piece_id=46MQMGAXKX2MFI2IXJL7ZWTBVXYQRM6XA7XD565K2F73AKNSTN6A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.852333363 (2h37m24.314425866s) [noise=true , address=77.90.31.12:4589 , piece_id=IRIEBUXYXWV2WHWNQDZT2ZPE75VHH4W3SURAYJ4MKMZVLIUUAUBQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853073563 (2h37m24.315166037s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853408290 (2h37m24.315500763s) [noise=true , address=45.91.116.11:4117 , piece_id=Q2P4XNRWOQEM257IMW253FIOTAWSPG3IEL55VTB2NHXSAQWAXH2A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853673696 (2h37m24.315766179s) [noise=true , address=45.91.116.5:4298 , piece_id=4T5H6GEBD5CWFHF7WKB4ISNKLF2ZS5DFVRZFWMQR7JUMWD6OFQBA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853967866 (2h37m24.316060349s) [noise=true , address=45.91.116.12:4227 , piece_id=IJTDTA3FDZCCVCCMWN3UWSNTWTSDU6UMY6F24GENRLVD3IXJ33WQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854047137 (2h37m24.31613961s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854304707 (2h37m24.31639718s) [noise=true , address=45.91.116.9:4262 , piece_id=LO2THA7C52KXZGEPEFXDSJOUBQDUIAJCO6UNQF7ZXZ7ETHN4KIOA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854435016 (2h37m24.316527499s) [noise=true , address=77.90.31.10:4519 , piece_id=V7GQEGTIC643FJXEIXRRX64APJQD4YKJFLKEHXFBWXMRUGQ6I4KA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854553952 (2h37m24.316646425s) [noise=true , address=145.79.227.5:4222 , piece_id=HE27KIBGHAZPFT5FUX7FL4KLMWRG3YBZ74N4Q4FGROSK2VFA5S3A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854819237 (2h37m24.31691172s) [noise=true , address=45.91.116.10:4082 , piece_id=YP64RUTIYQWN775JB527KG3KOJQMMIK6AJYRAJLPAIQBQH4NIZVA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854858622 (2h37m24.316951095s) [noise=true , address=45.91.116.12:4225 , piece_id=NF6W3OWZ7CD2XYLUPWAQX6EPGI753JCTD6GYGX6PSIIUQ5OD6SOA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.855251851 (2h37m24.317344324s) [noise=true , address=145.79.227.15:4876 , piece_id=Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.855651973 (2h37m24.317744456s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.856044019 (2h37m24.318136502s) [noise=true , address=145.79.227.19:4079 , piece_id=ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.856461354 (2h37m24.318553827s) [noise=true , address=145.79.227.14:4839 , piece_id=MPMNQZLCBFLMMBPYEQCJRJTTCZGZPZRGPDYW55PBM2I6V5RKXBAQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:40.657981922 (2h37m26.120074406s) [upload=2325]: Next returned nil segment Sep 23 16:08:40.658017650 (2h37m26.120110144s) [upload=2325]: Waiting for error group managing segments... Sep 23 16:08:40.965574730 (2h37m26.427667203s) [noise=true , address=45.91.116.5:4298 , piece_id=4T5H6GEBD5CWFHF7WKB4ISNKLF2ZS5DFVRZFWMQR7JUMWD6OFQBA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.017924353 (2h37m26.480016836s) [noise=true , address=45.91.116.9:4262 , piece_id=LO2THA7C52KXZGEPEFXDSJOUBQDUIAJCO6UNQF7ZXZ7ETHN4KIOA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.055061418 (2h37m26.517153891s) [noise=true , address=45.91.116.12:4225 , piece_id=NF6W3OWZ7CD2XYLUPWAQX6EPGI753JCTD6GYGX6PSIIUQ5OD6SOA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.059138873 (2h37m26.521231346s) [noise=true , address=77.90.31.10:4519 , piece_id=V7GQEGTIC643FJXEIXRRX64APJQD4YKJFLKEHXFBWXMRUGQ6I4KA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.068239702 (2h37m26.530332175s) [noise=true , address=45.91.116.11:4117 , piece_id=Q2P4XNRWOQEM257IMW253FIOTAWSPG3IEL55VTB2NHXSAQWAXH2A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.071035325 (2h37m26.533127798s) [noise=true , address=45.91.116.10:4082 , piece_id=YP64RUTIYQWN775JB527KG3KOJQMMIK6AJYRAJLPAIQBQH4NIZVA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.120390436 (2h37m26.582482919s) [noise=true , address=45.91.116.12:4227 , piece_id=IJTDTA3FDZCCVCCMWN3UWSNTWTSDU6UMY6F24GENRLVD3IXJ33WQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.262045750 (2h37m26.724138234s) [noise=true , address=77.90.31.12:4589 , piece_id=IRIEBUXYXWV2WHWNQDZT2ZPE75VHH4W3SURAYJ4MKMZVLIUUAUBQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:42.626787709 (2h37m28.088880182s) [noise=true , address=145.79.227.19:4079 , piece_id=ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:42.641124409 (2h37m28.103216882s) [noise=true , address=145.79.227.15:4876 , piece_id=Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.056725062 (2h37m28.518817545s) [noise=true , address=145.79.227.14:4839 , piece_id=MPMNQZLCBFLMMBPYEQCJRJTTCZGZPZRGPDYW55PBM2I6V5RKXBAQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.577366321 (2h37m29.039458804s) [noise=true , address=145.79.227.5:4222 , piece_id=HE27KIBGHAZPFT5FUX7FL4KLMWRG3YBZ74N4Q4FGROSK2VFA5S3A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.993428202 (2h37m29.455520675s) [noise=true , address=145.79.227.11:4731 , piece_id=46MQMGAXKX2MFI2IXJL7ZWTBVXYQRM6XA7XD565K2F73AKNSTN6A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.993452629 (2h37m29.455545102s) [seg_pos={0 0}, upload=2325]: Segment reached optimal threshold of 13 pieces. Sep 23 16:08:43.993624226 (2h37m29.455716709s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: expected piece hash; context canceled; EOF Sep 23 16:08:43.993637912 (2h37m29.455730395s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993678629 (2h37m29.455771112s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ): protocol: write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection Sep 23 16:08:43.993693137 (2h37m29.45578561s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993722223 (2h37m29.455814706s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Sep 23 16:08:43.993731370 (2h37m29.455823853s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993751087 (2h37m29.45584356s) [seg_pos={0 0}, upload=2325]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 16:08:43.993849845 (2h37m29.455942328s) [upload=2325]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 16:08:43.993858933 (2h37m29.455951416s) [upload=2325]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 16:08:44.305789411 (2h37m29.767881884s) [upload=2325]: Done uploading segments... Sep 23 16:08:44.305854725 (2h37m29.767947209s) [upload=2325]: Upload finished. err: <nil> Duration: 6.034416637s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.005925125s FAIL (2025-09-23 16:47:31 UTC) orbiter warning: context deadline exceeded Duration: 6.005956664s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.027730876s FAIL (2025-09-23 16:47:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA) context deadline exceeded; piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA) context deadline exceeded ======LOGS====== Sep 23 16:08:38.605894181 (2h37m24.067986664s) [upload=2325]: Starting upload Sep 23 16:08:38.606004200 (2h37m24.068096693s) [upload=2325]: Done starting upload Sep 23 16:08:38.606176218 (2h37m24.068268701s) [upload=2325]: Uploading segments... Sep 23 16:08:38.606277210 (2h37m24.068369683s) [upload=2325]: Got next segment. Inline: false Sep 23 16:08:38.606318859 (2h37m24.068411342s) [upload=2325]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 16:08:38.606324821 (2h37m24.068417304s) [upload=2325]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 16:08:38.852132441 (2h37m24.314224925s) [seg_pos={0 0}, upload=2325]: Begin upload segment... Sep 23 16:08:38.852225929 (2h37m24.314318402s) [seg_pos={0 0}, upload=2325]: Done begin upload segment. Sep 23 16:08:38.852268150 (2h37m24.314360623s) [noise=true , address=145.79.227.11:4731 , piece_id=46MQMGAXKX2MFI2IXJL7ZWTBVXYQRM6XA7XD565K2F73AKNSTN6A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.852333363 (2h37m24.314425866s) [noise=true , address=77.90.31.12:4589 , piece_id=IRIEBUXYXWV2WHWNQDZT2ZPE75VHH4W3SURAYJ4MKMZVLIUUAUBQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853073563 (2h37m24.315166037s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853408290 (2h37m24.315500763s) [noise=true , address=45.91.116.11:4117 , piece_id=Q2P4XNRWOQEM257IMW253FIOTAWSPG3IEL55VTB2NHXSAQWAXH2A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853673696 (2h37m24.315766179s) [noise=true , address=45.91.116.5:4298 , piece_id=4T5H6GEBD5CWFHF7WKB4ISNKLF2ZS5DFVRZFWMQR7JUMWD6OFQBA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853967866 (2h37m24.316060349s) [noise=true , address=45.91.116.12:4227 , piece_id=IJTDTA3FDZCCVCCMWN3UWSNTWTSDU6UMY6F24GENRLVD3IXJ33WQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854047137 (2h37m24.31613961s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854304707 (2h37m24.31639718s) [noise=true , address=45.91.116.9:4262 , piece_id=LO2THA7C52KXZGEPEFXDSJOUBQDUIAJCO6UNQF7ZXZ7ETHN4KIOA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854435016 (2h37m24.316527499s) [noise=true , address=77.90.31.10:4519 , piece_id=V7GQEGTIC643FJXEIXRRX64APJQD4YKJFLKEHXFBWXMRUGQ6I4KA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854553952 (2h37m24.316646425s) [noise=true , address=145.79.227.5:4222 , piece_id=HE27KIBGHAZPFT5FUX7FL4KLMWRG3YBZ74N4Q4FGROSK2VFA5S3A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854819237 (2h37m24.31691172s) [noise=true , address=45.91.116.10:4082 , piece_id=YP64RUTIYQWN775JB527KG3KOJQMMIK6AJYRAJLPAIQBQH4NIZVA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854858622 (2h37m24.316951095s) [noise=true , address=45.91.116.12:4225 , piece_id=NF6W3OWZ7CD2XYLUPWAQX6EPGI753JCTD6GYGX6PSIIUQ5OD6SOA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.855251851 (2h37m24.317344324s) [noise=true , address=145.79.227.15:4876 , piece_id=Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.855651973 (2h37m24.317744456s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.856044019 (2h37m24.318136502s) [noise=true , address=145.79.227.19:4079 , piece_id=ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.856461354 (2h37m24.318553827s) [noise=true , address=145.79.227.14:4839 , piece_id=MPMNQZLCBFLMMBPYEQCJRJTTCZGZPZRGPDYW55PBM2I6V5RKXBAQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:40.657981922 (2h37m26.120074406s) [upload=2325]: Next returned nil segment Sep 23 16:08:40.658017650 (2h37m26.120110144s) [upload=2325]: Waiting for error group managing segments... Sep 23 16:08:40.965574730 (2h37m26.427667203s) [noise=true , address=45.91.116.5:4298 , piece_id=4T5H6GEBD5CWFHF7WKB4ISNKLF2ZS5DFVRZFWMQR7JUMWD6OFQBA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.017924353 (2h37m26.480016836s) [noise=true , address=45.91.116.9:4262 , piece_id=LO2THA7C52KXZGEPEFXDSJOUBQDUIAJCO6UNQF7ZXZ7ETHN4KIOA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.055061418 (2h37m26.517153891s) [noise=true , address=45.91.116.12:4225 , piece_id=NF6W3OWZ7CD2XYLUPWAQX6EPGI753JCTD6GYGX6PSIIUQ5OD6SOA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.059138873 (2h37m26.521231346s) [noise=true , address=77.90.31.10:4519 , piece_id=V7GQEGTIC643FJXEIXRRX64APJQD4YKJFLKEHXFBWXMRUGQ6I4KA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.068239702 (2h37m26.530332175s) [noise=true , address=45.91.116.11:4117 , piece_id=Q2P4XNRWOQEM257IMW253FIOTAWSPG3IEL55VTB2NHXSAQWAXH2A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.071035325 (2h37m26.533127798s) [noise=true , address=45.91.116.10:4082 , piece_id=YP64RUTIYQWN775JB527KG3KOJQMMIK6AJYRAJLPAIQBQH4NIZVA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.120390436 (2h37m26.582482919s) [noise=true , address=45.91.116.12:4227 , piece_id=IJTDTA3FDZCCVCCMWN3UWSNTWTSDU6UMY6F24GENRLVD3IXJ33WQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.262045750 (2h37m26.724138234s) [noise=true , address=77.90.31.12:4589 , piece_id=IRIEBUXYXWV2WHWNQDZT2ZPE75VHH4W3SURAYJ4MKMZVLIUUAUBQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:42.626787709 (2h37m28.088880182s) [noise=true , address=145.79.227.19:4079 , piece_id=ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:42.641124409 (2h37m28.103216882s) [noise=true , address=145.79.227.15:4876 , piece_id=Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.056725062 (2h37m28.518817545s) [noise=true , address=145.79.227.14:4839 , piece_id=MPMNQZLCBFLMMBPYEQCJRJTTCZGZPZRGPDYW55PBM2I6V5RKXBAQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.577366321 (2h37m29.039458804s) [noise=true , address=145.79.227.5:4222 , piece_id=HE27KIBGHAZPFT5FUX7FL4KLMWRG3YBZ74N4Q4FGROSK2VFA5S3A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.993428202 (2h37m29.455520675s) [noise=true , address=145.79.227.11:4731 , piece_id=46MQMGAXKX2MFI2IXJL7ZWTBVXYQRM6XA7XD565K2F73AKNSTN6A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.993452629 (2h37m29.455545102s) [seg_pos={0 0}, upload=2325]: Segment reached optimal threshold of 13 pieces. Sep 23 16:08:43.993624226 (2h37m29.455716709s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: expected piece hash; context canceled; EOF Sep 23 16:08:43.993637912 (2h37m29.455730395s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993678629 (2h37m29.455771112s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ): protocol: write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection Sep 23 16:08:43.993693137 (2h37m29.45578561s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993722223 (2h37m29.455814706s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Sep 23 16:08:43.993731370 (2h37m29.455823853s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993751087 (2h37m29.45584356s) [seg_pos={0 0}, upload=2325]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 16:08:43.993849845 (2h37m29.455942328s) [upload=2325]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 16:08:43.993858933 (2h37m29.455951416s) [upload=2325]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 16:08:44.305789411 (2h37m29.767881884s) [upload=2325]: Done uploading segments... Sep 23 16:08:44.305854725 (2h37m29.767947209s) [upload=2325]: Upload finished. err: <nil> Duration: 6.027772184s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.321327245s FAIL (2025-09-23 16:47:10 UTC) orbiter warning: context deadline exceeded Duration: 6.321358705s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.084974909s FAIL (2025-09-23 16:46:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA) context deadline exceeded; piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA) context deadline exceeded ======LOGS====== Sep 23 16:08:38.605894181 (2h37m24.067986664s) [upload=2325]: Starting upload Sep 23 16:08:38.606004200 (2h37m24.068096693s) [upload=2325]: Done starting upload Sep 23 16:08:38.606176218 (2h37m24.068268701s) [upload=2325]: Uploading segments... Sep 23 16:08:38.606277210 (2h37m24.068369683s) [upload=2325]: Got next segment. Inline: false Sep 23 16:08:38.606318859 (2h37m24.068411342s) [upload=2325]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 16:08:38.606324821 (2h37m24.068417304s) [upload=2325]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 16:08:38.852132441 (2h37m24.314224925s) [seg_pos={0 0}, upload=2325]: Begin upload segment... Sep 23 16:08:38.852225929 (2h37m24.314318402s) [seg_pos={0 0}, upload=2325]: Done begin upload segment. Sep 23 16:08:38.852268150 (2h37m24.314360623s) [noise=true , address=145.79.227.11:4731 , piece_id=46MQMGAXKX2MFI2IXJL7ZWTBVXYQRM6XA7XD565K2F73AKNSTN6A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.852333363 (2h37m24.314425866s) [noise=true , address=77.90.31.12:4589 , piece_id=IRIEBUXYXWV2WHWNQDZT2ZPE75VHH4W3SURAYJ4MKMZVLIUUAUBQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853073563 (2h37m24.315166037s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853408290 (2h37m24.315500763s) [noise=true , address=45.91.116.11:4117 , piece_id=Q2P4XNRWOQEM257IMW253FIOTAWSPG3IEL55VTB2NHXSAQWAXH2A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853673696 (2h37m24.315766179s) [noise=true , address=45.91.116.5:4298 , piece_id=4T5H6GEBD5CWFHF7WKB4ISNKLF2ZS5DFVRZFWMQR7JUMWD6OFQBA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853967866 (2h37m24.316060349s) [noise=true , address=45.91.116.12:4227 , piece_id=IJTDTA3FDZCCVCCMWN3UWSNTWTSDU6UMY6F24GENRLVD3IXJ33WQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854047137 (2h37m24.31613961s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854304707 (2h37m24.31639718s) [noise=true , address=45.91.116.9:4262 , piece_id=LO2THA7C52KXZGEPEFXDSJOUBQDUIAJCO6UNQF7ZXZ7ETHN4KIOA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854435016 (2h37m24.316527499s) [noise=true , address=77.90.31.10:4519 , piece_id=V7GQEGTIC643FJXEIXRRX64APJQD4YKJFLKEHXFBWXMRUGQ6I4KA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854553952 (2h37m24.316646425s) [noise=true , address=145.79.227.5:4222 , piece_id=HE27KIBGHAZPFT5FUX7FL4KLMWRG3YBZ74N4Q4FGROSK2VFA5S3A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854819237 (2h37m24.31691172s) [noise=true , address=45.91.116.10:4082 , piece_id=YP64RUTIYQWN775JB527KG3KOJQMMIK6AJYRAJLPAIQBQH4NIZVA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854858622 (2h37m24.316951095s) [noise=true , address=45.91.116.12:4225 , piece_id=NF6W3OWZ7CD2XYLUPWAQX6EPGI753JCTD6GYGX6PSIIUQ5OD6SOA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.855251851 (2h37m24.317344324s) [noise=true , address=145.79.227.15:4876 , piece_id=Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.855651973 (2h37m24.317744456s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.856044019 (2h37m24.318136502s) [noise=true , address=145.79.227.19:4079 , piece_id=ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.856461354 (2h37m24.318553827s) [noise=true , address=145.79.227.14:4839 , piece_id=MPMNQZLCBFLMMBPYEQCJRJTTCZGZPZRGPDYW55PBM2I6V5RKXBAQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:40.657981922 (2h37m26.120074406s) [upload=2325]: Next returned nil segment Sep 23 16:08:40.658017650 (2h37m26.120110144s) [upload=2325]: Waiting for error group managing segments... Sep 23 16:08:40.965574730 (2h37m26.427667203s) [noise=true , address=45.91.116.5:4298 , piece_id=4T5H6GEBD5CWFHF7WKB4ISNKLF2ZS5DFVRZFWMQR7JUMWD6OFQBA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.017924353 (2h37m26.480016836s) [noise=true , address=45.91.116.9:4262 , piece_id=LO2THA7C52KXZGEPEFXDSJOUBQDUIAJCO6UNQF7ZXZ7ETHN4KIOA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.055061418 (2h37m26.517153891s) [noise=true , address=45.91.116.12:4225 , piece_id=NF6W3OWZ7CD2XYLUPWAQX6EPGI753JCTD6GYGX6PSIIUQ5OD6SOA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.059138873 (2h37m26.521231346s) [noise=true , address=77.90.31.10:4519 , piece_id=V7GQEGTIC643FJXEIXRRX64APJQD4YKJFLKEHXFBWXMRUGQ6I4KA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.068239702 (2h37m26.530332175s) [noise=true , address=45.91.116.11:4117 , piece_id=Q2P4XNRWOQEM257IMW253FIOTAWSPG3IEL55VTB2NHXSAQWAXH2A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.071035325 (2h37m26.533127798s) [noise=true , address=45.91.116.10:4082 , piece_id=YP64RUTIYQWN775JB527KG3KOJQMMIK6AJYRAJLPAIQBQH4NIZVA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.120390436 (2h37m26.582482919s) [noise=true , address=45.91.116.12:4227 , piece_id=IJTDTA3FDZCCVCCMWN3UWSNTWTSDU6UMY6F24GENRLVD3IXJ33WQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.262045750 (2h37m26.724138234s) [noise=true , address=77.90.31.12:4589 , piece_id=IRIEBUXYXWV2WHWNQDZT2ZPE75VHH4W3SURAYJ4MKMZVLIUUAUBQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:42.626787709 (2h37m28.088880182s) [noise=true , address=145.79.227.19:4079 , piece_id=ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:42.641124409 (2h37m28.103216882s) [noise=true , address=145.79.227.15:4876 , piece_id=Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.056725062 (2h37m28.518817545s) [noise=true , address=145.79.227.14:4839 , piece_id=MPMNQZLCBFLMMBPYEQCJRJTTCZGZPZRGPDYW55PBM2I6V5RKXBAQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.577366321 (2h37m29.039458804s) [noise=true , address=145.79.227.5:4222 , piece_id=HE27KIBGHAZPFT5FUX7FL4KLMWRG3YBZ74N4Q4FGROSK2VFA5S3A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.993428202 (2h37m29.455520675s) [noise=true , address=145.79.227.11:4731 , piece_id=46MQMGAXKX2MFI2IXJL7ZWTBVXYQRM6XA7XD565K2F73AKNSTN6A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.993452629 (2h37m29.455545102s) [seg_pos={0 0}, upload=2325]: Segment reached optimal threshold of 13 pieces. Sep 23 16:08:43.993624226 (2h37m29.455716709s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: expected piece hash; context canceled; EOF Sep 23 16:08:43.993637912 (2h37m29.455730395s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993678629 (2h37m29.455771112s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ): protocol: write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection Sep 23 16:08:43.993693137 (2h37m29.45578561s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993722223 (2h37m29.455814706s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Sep 23 16:08:43.993731370 (2h37m29.455823853s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993751087 (2h37m29.45584356s) [seg_pos={0 0}, upload=2325]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 16:08:43.993849845 (2h37m29.455942328s) [upload=2325]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 16:08:43.993858933 (2h37m29.455951416s) [upload=2325]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 16:08:44.305789411 (2h37m29.767881884s) [upload=2325]: Done uploading segments... Sep 23 16:08:44.305854725 (2h37m29.767947209s) [upload=2325]: Upload finished. err: <nil> Duration: 6.085013953s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.134263212s FAIL (2025-09-23 16:45:59 UTC) orbiter warning: context deadline exceeded Duration: 6.134285813s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.083100008s FAIL (2025-09-23 16:45:41 UTC) orbiter warning: context deadline exceeded Duration: 6.083159783s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.899824862s FAIL (2025-09-23 16:45:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA) context deadline exceeded; piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA) context deadline exceeded ======LOGS====== Sep 23 16:08:38.605894181 (2h37m24.067986664s) [upload=2325]: Starting upload Sep 23 16:08:38.606004200 (2h37m24.068096693s) [upload=2325]: Done starting upload Sep 23 16:08:38.606176218 (2h37m24.068268701s) [upload=2325]: Uploading segments... Sep 23 16:08:38.606277210 (2h37m24.068369683s) [upload=2325]: Got next segment. Inline: false Sep 23 16:08:38.606318859 (2h37m24.068411342s) [upload=2325]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 16:08:38.606324821 (2h37m24.068417304s) [upload=2325]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 16:08:38.852132441 (2h37m24.314224925s) [seg_pos={0 0}, upload=2325]: Begin upload segment... Sep 23 16:08:38.852225929 (2h37m24.314318402s) [seg_pos={0 0}, upload=2325]: Done begin upload segment. Sep 23 16:08:38.852268150 (2h37m24.314360623s) [noise=true , address=145.79.227.11:4731 , piece_id=46MQMGAXKX2MFI2IXJL7ZWTBVXYQRM6XA7XD565K2F73AKNSTN6A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.852333363 (2h37m24.314425866s) [noise=true , address=77.90.31.12:4589 , piece_id=IRIEBUXYXWV2WHWNQDZT2ZPE75VHH4W3SURAYJ4MKMZVLIUUAUBQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853073563 (2h37m24.315166037s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853408290 (2h37m24.315500763s) [noise=true , address=45.91.116.11:4117 , piece_id=Q2P4XNRWOQEM257IMW253FIOTAWSPG3IEL55VTB2NHXSAQWAXH2A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853673696 (2h37m24.315766179s) [noise=true , address=45.91.116.5:4298 , piece_id=4T5H6GEBD5CWFHF7WKB4ISNKLF2ZS5DFVRZFWMQR7JUMWD6OFQBA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853967866 (2h37m24.316060349s) [noise=true , address=45.91.116.12:4227 , piece_id=IJTDTA3FDZCCVCCMWN3UWSNTWTSDU6UMY6F24GENRLVD3IXJ33WQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854047137 (2h37m24.31613961s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854304707 (2h37m24.31639718s) [noise=true , address=45.91.116.9:4262 , piece_id=LO2THA7C52KXZGEPEFXDSJOUBQDUIAJCO6UNQF7ZXZ7ETHN4KIOA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854435016 (2h37m24.316527499s) [noise=true , address=77.90.31.10:4519 , piece_id=V7GQEGTIC643FJXEIXRRX64APJQD4YKJFLKEHXFBWXMRUGQ6I4KA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854553952 (2h37m24.316646425s) [noise=true , address=145.79.227.5:4222 , piece_id=HE27KIBGHAZPFT5FUX7FL4KLMWRG3YBZ74N4Q4FGROSK2VFA5S3A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854819237 (2h37m24.31691172s) [noise=true , address=45.91.116.10:4082 , piece_id=YP64RUTIYQWN775JB527KG3KOJQMMIK6AJYRAJLPAIQBQH4NIZVA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854858622 (2h37m24.316951095s) [noise=true , address=45.91.116.12:4225 , piece_id=NF6W3OWZ7CD2XYLUPWAQX6EPGI753JCTD6GYGX6PSIIUQ5OD6SOA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.855251851 (2h37m24.317344324s) [noise=true , address=145.79.227.15:4876 , piece_id=Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.855651973 (2h37m24.317744456s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.856044019 (2h37m24.318136502s) [noise=true , address=145.79.227.19:4079 , piece_id=ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.856461354 (2h37m24.318553827s) [noise=true , address=145.79.227.14:4839 , piece_id=MPMNQZLCBFLMMBPYEQCJRJTTCZGZPZRGPDYW55PBM2I6V5RKXBAQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:40.657981922 (2h37m26.120074406s) [upload=2325]: Next returned nil segment Sep 23 16:08:40.658017650 (2h37m26.120110144s) [upload=2325]: Waiting for error group managing segments... Sep 23 16:08:40.965574730 (2h37m26.427667203s) [noise=true , address=45.91.116.5:4298 , piece_id=4T5H6GEBD5CWFHF7WKB4ISNKLF2ZS5DFVRZFWMQR7JUMWD6OFQBA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.017924353 (2h37m26.480016836s) [noise=true , address=45.91.116.9:4262 , piece_id=LO2THA7C52KXZGEPEFXDSJOUBQDUIAJCO6UNQF7ZXZ7ETHN4KIOA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.055061418 (2h37m26.517153891s) [noise=true , address=45.91.116.12:4225 , piece_id=NF6W3OWZ7CD2XYLUPWAQX6EPGI753JCTD6GYGX6PSIIUQ5OD6SOA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.059138873 (2h37m26.521231346s) [noise=true , address=77.90.31.10:4519 , piece_id=V7GQEGTIC643FJXEIXRRX64APJQD4YKJFLKEHXFBWXMRUGQ6I4KA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.068239702 (2h37m26.530332175s) [noise=true , address=45.91.116.11:4117 , piece_id=Q2P4XNRWOQEM257IMW253FIOTAWSPG3IEL55VTB2NHXSAQWAXH2A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.071035325 (2h37m26.533127798s) [noise=true , address=45.91.116.10:4082 , piece_id=YP64RUTIYQWN775JB527KG3KOJQMMIK6AJYRAJLPAIQBQH4NIZVA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.120390436 (2h37m26.582482919s) [noise=true , address=45.91.116.12:4227 , piece_id=IJTDTA3FDZCCVCCMWN3UWSNTWTSDU6UMY6F24GENRLVD3IXJ33WQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.262045750 (2h37m26.724138234s) [noise=true , address=77.90.31.12:4589 , piece_id=IRIEBUXYXWV2WHWNQDZT2ZPE75VHH4W3SURAYJ4MKMZVLIUUAUBQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:42.626787709 (2h37m28.088880182s) [noise=true , address=145.79.227.19:4079 , piece_id=ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:42.641124409 (2h37m28.103216882s) [noise=true , address=145.79.227.15:4876 , piece_id=Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.056725062 (2h37m28.518817545s) [noise=true , address=145.79.227.14:4839 , piece_id=MPMNQZLCBFLMMBPYEQCJRJTTCZGZPZRGPDYW55PBM2I6V5RKXBAQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.577366321 (2h37m29.039458804s) [noise=true , address=145.79.227.5:4222 , piece_id=HE27KIBGHAZPFT5FUX7FL4KLMWRG3YBZ74N4Q4FGROSK2VFA5S3A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.993428202 (2h37m29.455520675s) [noise=true , address=145.79.227.11:4731 , piece_id=46MQMGAXKX2MFI2IXJL7ZWTBVXYQRM6XA7XD565K2F73AKNSTN6A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.993452629 (2h37m29.455545102s) [seg_pos={0 0}, upload=2325]: Segment reached optimal threshold of 13 pieces. Sep 23 16:08:43.993624226 (2h37m29.455716709s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: expected piece hash; context canceled; EOF Sep 23 16:08:43.993637912 (2h37m29.455730395s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993678629 (2h37m29.455771112s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ): protocol: write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection Sep 23 16:08:43.993693137 (2h37m29.45578561s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993722223 (2h37m29.455814706s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Sep 23 16:08:43.993731370 (2h37m29.455823853s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993751087 (2h37m29.45584356s) [seg_pos={0 0}, upload=2325]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 16:08:43.993849845 (2h37m29.455942328s) [upload=2325]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 16:08:43.993858933 (2h37m29.455951416s) [upload=2325]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 16:08:44.305789411 (2h37m29.767881884s) [upload=2325]: Done uploading segments... Sep 23 16:08:44.305854725 (2h37m29.767947209s) [upload=2325]: Upload finished. err: <nil> Duration: 5.899856861s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.100502907s FAIL (2025-09-23 16:45:06 UTC) orbiter warning: context deadline exceeded Duration: 6.100529448s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.886511489s FAIL (2025-09-23 16:44:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA) context deadline exceeded; piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA) context deadline exceeded ======LOGS====== Sep 23 16:08:38.605894181 (2h37m24.067986664s) [upload=2325]: Starting upload Sep 23 16:08:38.606004200 (2h37m24.068096693s) [upload=2325]: Done starting upload Sep 23 16:08:38.606176218 (2h37m24.068268701s) [upload=2325]: Uploading segments... Sep 23 16:08:38.606277210 (2h37m24.068369683s) [upload=2325]: Got next segment. Inline: false Sep 23 16:08:38.606318859 (2h37m24.068411342s) [upload=2325]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 16:08:38.606324821 (2h37m24.068417304s) [upload=2325]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 16:08:38.852132441 (2h37m24.314224925s) [seg_pos={0 0}, upload=2325]: Begin upload segment... Sep 23 16:08:38.852225929 (2h37m24.314318402s) [seg_pos={0 0}, upload=2325]: Done begin upload segment. Sep 23 16:08:38.852268150 (2h37m24.314360623s) [noise=true , address=145.79.227.11:4731 , piece_id=46MQMGAXKX2MFI2IXJL7ZWTBVXYQRM6XA7XD565K2F73AKNSTN6A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.852333363 (2h37m24.314425866s) [noise=true , address=77.90.31.12:4589 , piece_id=IRIEBUXYXWV2WHWNQDZT2ZPE75VHH4W3SURAYJ4MKMZVLIUUAUBQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853073563 (2h37m24.315166037s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853408290 (2h37m24.315500763s) [noise=true , address=45.91.116.11:4117 , piece_id=Q2P4XNRWOQEM257IMW253FIOTAWSPG3IEL55VTB2NHXSAQWAXH2A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853673696 (2h37m24.315766179s) [noise=true , address=45.91.116.5:4298 , piece_id=4T5H6GEBD5CWFHF7WKB4ISNKLF2ZS5DFVRZFWMQR7JUMWD6OFQBA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853967866 (2h37m24.316060349s) [noise=true , address=45.91.116.12:4227 , piece_id=IJTDTA3FDZCCVCCMWN3UWSNTWTSDU6UMY6F24GENRLVD3IXJ33WQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854047137 (2h37m24.31613961s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854304707 (2h37m24.31639718s) [noise=true , address=45.91.116.9:4262 , piece_id=LO2THA7C52KXZGEPEFXDSJOUBQDUIAJCO6UNQF7ZXZ7ETHN4KIOA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854435016 (2h37m24.316527499s) [noise=true , address=77.90.31.10:4519 , piece_id=V7GQEGTIC643FJXEIXRRX64APJQD4YKJFLKEHXFBWXMRUGQ6I4KA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854553952 (2h37m24.316646425s) [noise=true , address=145.79.227.5:4222 , piece_id=HE27KIBGHAZPFT5FUX7FL4KLMWRG3YBZ74N4Q4FGROSK2VFA5S3A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854819237 (2h37m24.31691172s) [noise=true , address=45.91.116.10:4082 , piece_id=YP64RUTIYQWN775JB527KG3KOJQMMIK6AJYRAJLPAIQBQH4NIZVA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854858622 (2h37m24.316951095s) [noise=true , address=45.91.116.12:4225 , piece_id=NF6W3OWZ7CD2XYLUPWAQX6EPGI753JCTD6GYGX6PSIIUQ5OD6SOA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.855251851 (2h37m24.317344324s) [noise=true , address=145.79.227.15:4876 , piece_id=Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.855651973 (2h37m24.317744456s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.856044019 (2h37m24.318136502s) [noise=true , address=145.79.227.19:4079 , piece_id=ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.856461354 (2h37m24.318553827s) [noise=true , address=145.79.227.14:4839 , piece_id=MPMNQZLCBFLMMBPYEQCJRJTTCZGZPZRGPDYW55PBM2I6V5RKXBAQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:40.657981922 (2h37m26.120074406s) [upload=2325]: Next returned nil segment Sep 23 16:08:40.658017650 (2h37m26.120110144s) [upload=2325]: Waiting for error group managing segments... Sep 23 16:08:40.965574730 (2h37m26.427667203s) [noise=true , address=45.91.116.5:4298 , piece_id=4T5H6GEBD5CWFHF7WKB4ISNKLF2ZS5DFVRZFWMQR7JUMWD6OFQBA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.017924353 (2h37m26.480016836s) [noise=true , address=45.91.116.9:4262 , piece_id=LO2THA7C52KXZGEPEFXDSJOUBQDUIAJCO6UNQF7ZXZ7ETHN4KIOA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.055061418 (2h37m26.517153891s) [noise=true , address=45.91.116.12:4225 , piece_id=NF6W3OWZ7CD2XYLUPWAQX6EPGI753JCTD6GYGX6PSIIUQ5OD6SOA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.059138873 (2h37m26.521231346s) [noise=true , address=77.90.31.10:4519 , piece_id=V7GQEGTIC643FJXEIXRRX64APJQD4YKJFLKEHXFBWXMRUGQ6I4KA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.068239702 (2h37m26.530332175s) [noise=true , address=45.91.116.11:4117 , piece_id=Q2P4XNRWOQEM257IMW253FIOTAWSPG3IEL55VTB2NHXSAQWAXH2A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.071035325 (2h37m26.533127798s) [noise=true , address=45.91.116.10:4082 , piece_id=YP64RUTIYQWN775JB527KG3KOJQMMIK6AJYRAJLPAIQBQH4NIZVA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.120390436 (2h37m26.582482919s) [noise=true , address=45.91.116.12:4227 , piece_id=IJTDTA3FDZCCVCCMWN3UWSNTWTSDU6UMY6F24GENRLVD3IXJ33WQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.262045750 (2h37m26.724138234s) [noise=true , address=77.90.31.12:4589 , piece_id=IRIEBUXYXWV2WHWNQDZT2ZPE75VHH4W3SURAYJ4MKMZVLIUUAUBQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:42.626787709 (2h37m28.088880182s) [noise=true , address=145.79.227.19:4079 , piece_id=ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:42.641124409 (2h37m28.103216882s) [noise=true , address=145.79.227.15:4876 , piece_id=Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.056725062 (2h37m28.518817545s) [noise=true , address=145.79.227.14:4839 , piece_id=MPMNQZLCBFLMMBPYEQCJRJTTCZGZPZRGPDYW55PBM2I6V5RKXBAQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.577366321 (2h37m29.039458804s) [noise=true , address=145.79.227.5:4222 , piece_id=HE27KIBGHAZPFT5FUX7FL4KLMWRG3YBZ74N4Q4FGROSK2VFA5S3A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.993428202 (2h37m29.455520675s) [noise=true , address=145.79.227.11:4731 , piece_id=46MQMGAXKX2MFI2IXJL7ZWTBVXYQRM6XA7XD565K2F73AKNSTN6A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.993452629 (2h37m29.455545102s) [seg_pos={0 0}, upload=2325]: Segment reached optimal threshold of 13 pieces. Sep 23 16:08:43.993624226 (2h37m29.455716709s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: expected piece hash; context canceled; EOF Sep 23 16:08:43.993637912 (2h37m29.455730395s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993678629 (2h37m29.455771112s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ): protocol: write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection Sep 23 16:08:43.993693137 (2h37m29.45578561s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993722223 (2h37m29.455814706s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Sep 23 16:08:43.993731370 (2h37m29.455823853s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993751087 (2h37m29.45584356s) [seg_pos={0 0}, upload=2325]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 16:08:43.993849845 (2h37m29.455942328s) [upload=2325]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 16:08:43.993858933 (2h37m29.455951416s) [upload=2325]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 16:08:44.305789411 (2h37m29.767881884s) [upload=2325]: Done uploading segments... Sep 23 16:08:44.305854725 (2h37m29.767947209s) [upload=2325]: Upload finished. err: <nil> Duration: 5.88654338s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.037572355s FAIL (2025-09-23 16:43:58 UTC) orbiter warning: context deadline exceeded Duration: 6.0376483s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.988618046s FAIL (2025-09-23 16:43:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA) context deadline exceeded; piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA) context deadline exceeded ======LOGS====== Sep 23 16:08:38.605894181 (2h37m24.067986664s) [upload=2325]: Starting upload Sep 23 16:08:38.606004200 (2h37m24.068096693s) [upload=2325]: Done starting upload Sep 23 16:08:38.606176218 (2h37m24.068268701s) [upload=2325]: Uploading segments... Sep 23 16:08:38.606277210 (2h37m24.068369683s) [upload=2325]: Got next segment. Inline: false Sep 23 16:08:38.606318859 (2h37m24.068411342s) [upload=2325]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 16:08:38.606324821 (2h37m24.068417304s) [upload=2325]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 16:08:38.852132441 (2h37m24.314224925s) [seg_pos={0 0}, upload=2325]: Begin upload segment... Sep 23 16:08:38.852225929 (2h37m24.314318402s) [seg_pos={0 0}, upload=2325]: Done begin upload segment. Sep 23 16:08:38.852268150 (2h37m24.314360623s) [noise=true , address=145.79.227.11:4731 , piece_id=46MQMGAXKX2MFI2IXJL7ZWTBVXYQRM6XA7XD565K2F73AKNSTN6A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.852333363 (2h37m24.314425866s) [noise=true , address=77.90.31.12:4589 , piece_id=IRIEBUXYXWV2WHWNQDZT2ZPE75VHH4W3SURAYJ4MKMZVLIUUAUBQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853073563 (2h37m24.315166037s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853408290 (2h37m24.315500763s) [noise=true , address=45.91.116.11:4117 , piece_id=Q2P4XNRWOQEM257IMW253FIOTAWSPG3IEL55VTB2NHXSAQWAXH2A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853673696 (2h37m24.315766179s) [noise=true , address=45.91.116.5:4298 , piece_id=4T5H6GEBD5CWFHF7WKB4ISNKLF2ZS5DFVRZFWMQR7JUMWD6OFQBA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853967866 (2h37m24.316060349s) [noise=true , address=45.91.116.12:4227 , piece_id=IJTDTA3FDZCCVCCMWN3UWSNTWTSDU6UMY6F24GENRLVD3IXJ33WQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854047137 (2h37m24.31613961s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854304707 (2h37m24.31639718s) [noise=true , address=45.91.116.9:4262 , piece_id=LO2THA7C52KXZGEPEFXDSJOUBQDUIAJCO6UNQF7ZXZ7ETHN4KIOA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854435016 (2h37m24.316527499s) [noise=true , address=77.90.31.10:4519 , piece_id=V7GQEGTIC643FJXEIXRRX64APJQD4YKJFLKEHXFBWXMRUGQ6I4KA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854553952 (2h37m24.316646425s) [noise=true , address=145.79.227.5:4222 , piece_id=HE27KIBGHAZPFT5FUX7FL4KLMWRG3YBZ74N4Q4FGROSK2VFA5S3A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854819237 (2h37m24.31691172s) [noise=true , address=45.91.116.10:4082 , piece_id=YP64RUTIYQWN775JB527KG3KOJQMMIK6AJYRAJLPAIQBQH4NIZVA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854858622 (2h37m24.316951095s) [noise=true , address=45.91.116.12:4225 , piece_id=NF6W3OWZ7CD2XYLUPWAQX6EPGI753JCTD6GYGX6PSIIUQ5OD6SOA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.855251851 (2h37m24.317344324s) [noise=true , address=145.79.227.15:4876 , piece_id=Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.855651973 (2h37m24.317744456s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.856044019 (2h37m24.318136502s) [noise=true , address=145.79.227.19:4079 , piece_id=ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.856461354 (2h37m24.318553827s) [noise=true , address=145.79.227.14:4839 , piece_id=MPMNQZLCBFLMMBPYEQCJRJTTCZGZPZRGPDYW55PBM2I6V5RKXBAQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:40.657981922 (2h37m26.120074406s) [upload=2325]: Next returned nil segment Sep 23 16:08:40.658017650 (2h37m26.120110144s) [upload=2325]: Waiting for error group managing segments... Sep 23 16:08:40.965574730 (2h37m26.427667203s) [noise=true , address=45.91.116.5:4298 , piece_id=4T5H6GEBD5CWFHF7WKB4ISNKLF2ZS5DFVRZFWMQR7JUMWD6OFQBA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.017924353 (2h37m26.480016836s) [noise=true , address=45.91.116.9:4262 , piece_id=LO2THA7C52KXZGEPEFXDSJOUBQDUIAJCO6UNQF7ZXZ7ETHN4KIOA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.055061418 (2h37m26.517153891s) [noise=true , address=45.91.116.12:4225 , piece_id=NF6W3OWZ7CD2XYLUPWAQX6EPGI753JCTD6GYGX6PSIIUQ5OD6SOA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.059138873 (2h37m26.521231346s) [noise=true , address=77.90.31.10:4519 , piece_id=V7GQEGTIC643FJXEIXRRX64APJQD4YKJFLKEHXFBWXMRUGQ6I4KA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.068239702 (2h37m26.530332175s) [noise=true , address=45.91.116.11:4117 , piece_id=Q2P4XNRWOQEM257IMW253FIOTAWSPG3IEL55VTB2NHXSAQWAXH2A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.071035325 (2h37m26.533127798s) [noise=true , address=45.91.116.10:4082 , piece_id=YP64RUTIYQWN775JB527KG3KOJQMMIK6AJYRAJLPAIQBQH4NIZVA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.120390436 (2h37m26.582482919s) [noise=true , address=45.91.116.12:4227 , piece_id=IJTDTA3FDZCCVCCMWN3UWSNTWTSDU6UMY6F24GENRLVD3IXJ33WQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.262045750 (2h37m26.724138234s) [noise=true , address=77.90.31.12:4589 , piece_id=IRIEBUXYXWV2WHWNQDZT2ZPE75VHH4W3SURAYJ4MKMZVLIUUAUBQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:42.626787709 (2h37m28.088880182s) [noise=true , address=145.79.227.19:4079 , piece_id=ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:42.641124409 (2h37m28.103216882s) [noise=true , address=145.79.227.15:4876 , piece_id=Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.056725062 (2h37m28.518817545s) [noise=true , address=145.79.227.14:4839 , piece_id=MPMNQZLCBFLMMBPYEQCJRJTTCZGZPZRGPDYW55PBM2I6V5RKXBAQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.577366321 (2h37m29.039458804s) [noise=true , address=145.79.227.5:4222 , piece_id=HE27KIBGHAZPFT5FUX7FL4KLMWRG3YBZ74N4Q4FGROSK2VFA5S3A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.993428202 (2h37m29.455520675s) [noise=true , address=145.79.227.11:4731 , piece_id=46MQMGAXKX2MFI2IXJL7ZWTBVXYQRM6XA7XD565K2F73AKNSTN6A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.993452629 (2h37m29.455545102s) [seg_pos={0 0}, upload=2325]: Segment reached optimal threshold of 13 pieces. Sep 23 16:08:43.993624226 (2h37m29.455716709s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: expected piece hash; context canceled; EOF Sep 23 16:08:43.993637912 (2h37m29.455730395s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993678629 (2h37m29.455771112s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ): protocol: write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection Sep 23 16:08:43.993693137 (2h37m29.45578561s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993722223 (2h37m29.455814706s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Sep 23 16:08:43.993731370 (2h37m29.455823853s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993751087 (2h37m29.45584356s) [seg_pos={0 0}, upload=2325]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 16:08:43.993849845 (2h37m29.455942328s) [upload=2325]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 16:08:43.993858933 (2h37m29.455951416s) [upload=2325]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 16:08:44.305789411 (2h37m29.767881884s) [upload=2325]: Done uploading segments... Sep 23 16:08:44.305854725 (2h37m29.767947209s) [upload=2325]: Upload finished. err: <nil> Duration: 5.988649956s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.35370637s FAIL (2025-09-23 16:43:25 UTC) orbiter warning: context deadline exceeded Duration: 6.35373291s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.963081743s FAIL (2025-09-23 16:42:38 UTC) orbiter warning: context deadline exceeded Duration: 5.963106771s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.846393638s FAIL (2025-09-23 16:42:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA) context deadline exceeded; piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA) context deadline exceeded ======LOGS====== Sep 23 16:08:38.605894181 (2h37m24.067986664s) [upload=2325]: Starting upload Sep 23 16:08:38.606004200 (2h37m24.068096693s) [upload=2325]: Done starting upload Sep 23 16:08:38.606176218 (2h37m24.068268701s) [upload=2325]: Uploading segments... Sep 23 16:08:38.606277210 (2h37m24.068369683s) [upload=2325]: Got next segment. Inline: false Sep 23 16:08:38.606318859 (2h37m24.068411342s) [upload=2325]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 16:08:38.606324821 (2h37m24.068417304s) [upload=2325]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 16:08:38.852132441 (2h37m24.314224925s) [seg_pos={0 0}, upload=2325]: Begin upload segment... Sep 23 16:08:38.852225929 (2h37m24.314318402s) [seg_pos={0 0}, upload=2325]: Done begin upload segment. Sep 23 16:08:38.852268150 (2h37m24.314360623s) [noise=true , address=145.79.227.11:4731 , piece_id=46MQMGAXKX2MFI2IXJL7ZWTBVXYQRM6XA7XD565K2F73AKNSTN6A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.852333363 (2h37m24.314425866s) [noise=true , address=77.90.31.12:4589 , piece_id=IRIEBUXYXWV2WHWNQDZT2ZPE75VHH4W3SURAYJ4MKMZVLIUUAUBQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853073563 (2h37m24.315166037s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853408290 (2h37m24.315500763s) [noise=true , address=45.91.116.11:4117 , piece_id=Q2P4XNRWOQEM257IMW253FIOTAWSPG3IEL55VTB2NHXSAQWAXH2A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853673696 (2h37m24.315766179s) [noise=true , address=45.91.116.5:4298 , piece_id=4T5H6GEBD5CWFHF7WKB4ISNKLF2ZS5DFVRZFWMQR7JUMWD6OFQBA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853967866 (2h37m24.316060349s) [noise=true , address=45.91.116.12:4227 , piece_id=IJTDTA3FDZCCVCCMWN3UWSNTWTSDU6UMY6F24GENRLVD3IXJ33WQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854047137 (2h37m24.31613961s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854304707 (2h37m24.31639718s) [noise=true , address=45.91.116.9:4262 , piece_id=LO2THA7C52KXZGEPEFXDSJOUBQDUIAJCO6UNQF7ZXZ7ETHN4KIOA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854435016 (2h37m24.316527499s) [noise=true , address=77.90.31.10:4519 , piece_id=V7GQEGTIC643FJXEIXRRX64APJQD4YKJFLKEHXFBWXMRUGQ6I4KA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854553952 (2h37m24.316646425s) [noise=true , address=145.79.227.5:4222 , piece_id=HE27KIBGHAZPFT5FUX7FL4KLMWRG3YBZ74N4Q4FGROSK2VFA5S3A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854819237 (2h37m24.31691172s) [noise=true , address=45.91.116.10:4082 , piece_id=YP64RUTIYQWN775JB527KG3KOJQMMIK6AJYRAJLPAIQBQH4NIZVA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854858622 (2h37m24.316951095s) [noise=true , address=45.91.116.12:4225 , piece_id=NF6W3OWZ7CD2XYLUPWAQX6EPGI753JCTD6GYGX6PSIIUQ5OD6SOA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.855251851 (2h37m24.317344324s) [noise=true , address=145.79.227.15:4876 , piece_id=Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.855651973 (2h37m24.317744456s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.856044019 (2h37m24.318136502s) [noise=true , address=145.79.227.19:4079 , piece_id=ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.856461354 (2h37m24.318553827s) [noise=true , address=145.79.227.14:4839 , piece_id=MPMNQZLCBFLMMBPYEQCJRJTTCZGZPZRGPDYW55PBM2I6V5RKXBAQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:40.657981922 (2h37m26.120074406s) [upload=2325]: Next returned nil segment Sep 23 16:08:40.658017650 (2h37m26.120110144s) [upload=2325]: Waiting for error group managing segments... Sep 23 16:08:40.965574730 (2h37m26.427667203s) [noise=true , address=45.91.116.5:4298 , piece_id=4T5H6GEBD5CWFHF7WKB4ISNKLF2ZS5DFVRZFWMQR7JUMWD6OFQBA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.017924353 (2h37m26.480016836s) [noise=true , address=45.91.116.9:4262 , piece_id=LO2THA7C52KXZGEPEFXDSJOUBQDUIAJCO6UNQF7ZXZ7ETHN4KIOA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.055061418 (2h37m26.517153891s) [noise=true , address=45.91.116.12:4225 , piece_id=NF6W3OWZ7CD2XYLUPWAQX6EPGI753JCTD6GYGX6PSIIUQ5OD6SOA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.059138873 (2h37m26.521231346s) [noise=true , address=77.90.31.10:4519 , piece_id=V7GQEGTIC643FJXEIXRRX64APJQD4YKJFLKEHXFBWXMRUGQ6I4KA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.068239702 (2h37m26.530332175s) [noise=true , address=45.91.116.11:4117 , piece_id=Q2P4XNRWOQEM257IMW253FIOTAWSPG3IEL55VTB2NHXSAQWAXH2A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.071035325 (2h37m26.533127798s) [noise=true , address=45.91.116.10:4082 , piece_id=YP64RUTIYQWN775JB527KG3KOJQMMIK6AJYRAJLPAIQBQH4NIZVA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.120390436 (2h37m26.582482919s) [noise=true , address=45.91.116.12:4227 , piece_id=IJTDTA3FDZCCVCCMWN3UWSNTWTSDU6UMY6F24GENRLVD3IXJ33WQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.262045750 (2h37m26.724138234s) [noise=true , address=77.90.31.12:4589 , piece_id=IRIEBUXYXWV2WHWNQDZT2ZPE75VHH4W3SURAYJ4MKMZVLIUUAUBQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:42.626787709 (2h37m28.088880182s) [noise=true , address=145.79.227.19:4079 , piece_id=ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:42.641124409 (2h37m28.103216882s) [noise=true , address=145.79.227.15:4876 , piece_id=Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.056725062 (2h37m28.518817545s) [noise=true , address=145.79.227.14:4839 , piece_id=MPMNQZLCBFLMMBPYEQCJRJTTCZGZPZRGPDYW55PBM2I6V5RKXBAQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.577366321 (2h37m29.039458804s) [noise=true , address=145.79.227.5:4222 , piece_id=HE27KIBGHAZPFT5FUX7FL4KLMWRG3YBZ74N4Q4FGROSK2VFA5S3A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.993428202 (2h37m29.455520675s) [noise=true , address=145.79.227.11:4731 , piece_id=46MQMGAXKX2MFI2IXJL7ZWTBVXYQRM6XA7XD565K2F73AKNSTN6A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.993452629 (2h37m29.455545102s) [seg_pos={0 0}, upload=2325]: Segment reached optimal threshold of 13 pieces. Sep 23 16:08:43.993624226 (2h37m29.455716709s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: expected piece hash; context canceled; EOF Sep 23 16:08:43.993637912 (2h37m29.455730395s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993678629 (2h37m29.455771112s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ): protocol: write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection Sep 23 16:08:43.993693137 (2h37m29.45578561s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993722223 (2h37m29.455814706s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Sep 23 16:08:43.993731370 (2h37m29.455823853s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993751087 (2h37m29.45584356s) [seg_pos={0 0}, upload=2325]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 16:08:43.993849845 (2h37m29.455942328s) [upload=2325]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 16:08:43.993858933 (2h37m29.455951416s) [upload=2325]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 16:08:44.305789411 (2h37m29.767881884s) [upload=2325]: Done uploading segments... Sep 23 16:08:44.305854725 (2h37m29.767947209s) [upload=2325]: Upload finished. err: <nil> Duration: 5.846445297s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.600250328s FAIL (2025-09-23 16:42:06 UTC) orbiter warning: context deadline exceeded Duration: 5.60027811s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.908105117s FAIL (2025-09-23 16:41:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA) context deadline exceeded; piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA) context deadline exceeded ======LOGS====== Sep 23 16:08:38.605894181 (2h37m24.067986664s) [upload=2325]: Starting upload Sep 23 16:08:38.606004200 (2h37m24.068096693s) [upload=2325]: Done starting upload Sep 23 16:08:38.606176218 (2h37m24.068268701s) [upload=2325]: Uploading segments... Sep 23 16:08:38.606277210 (2h37m24.068369683s) [upload=2325]: Got next segment. Inline: false Sep 23 16:08:38.606318859 (2h37m24.068411342s) [upload=2325]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 16:08:38.606324821 (2h37m24.068417304s) [upload=2325]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 16:08:38.852132441 (2h37m24.314224925s) [seg_pos={0 0}, upload=2325]: Begin upload segment... Sep 23 16:08:38.852225929 (2h37m24.314318402s) [seg_pos={0 0}, upload=2325]: Done begin upload segment. Sep 23 16:08:38.852268150 (2h37m24.314360623s) [noise=true , address=145.79.227.11:4731 , piece_id=46MQMGAXKX2MFI2IXJL7ZWTBVXYQRM6XA7XD565K2F73AKNSTN6A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.852333363 (2h37m24.314425866s) [noise=true , address=77.90.31.12:4589 , piece_id=IRIEBUXYXWV2WHWNQDZT2ZPE75VHH4W3SURAYJ4MKMZVLIUUAUBQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853073563 (2h37m24.315166037s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853408290 (2h37m24.315500763s) [noise=true , address=45.91.116.11:4117 , piece_id=Q2P4XNRWOQEM257IMW253FIOTAWSPG3IEL55VTB2NHXSAQWAXH2A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853673696 (2h37m24.315766179s) [noise=true , address=45.91.116.5:4298 , piece_id=4T5H6GEBD5CWFHF7WKB4ISNKLF2ZS5DFVRZFWMQR7JUMWD6OFQBA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853967866 (2h37m24.316060349s) [noise=true , address=45.91.116.12:4227 , piece_id=IJTDTA3FDZCCVCCMWN3UWSNTWTSDU6UMY6F24GENRLVD3IXJ33WQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854047137 (2h37m24.31613961s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854304707 (2h37m24.31639718s) [noise=true , address=45.91.116.9:4262 , piece_id=LO2THA7C52KXZGEPEFXDSJOUBQDUIAJCO6UNQF7ZXZ7ETHN4KIOA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854435016 (2h37m24.316527499s) [noise=true , address=77.90.31.10:4519 , piece_id=V7GQEGTIC643FJXEIXRRX64APJQD4YKJFLKEHXFBWXMRUGQ6I4KA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854553952 (2h37m24.316646425s) [noise=true , address=145.79.227.5:4222 , piece_id=HE27KIBGHAZPFT5FUX7FL4KLMWRG3YBZ74N4Q4FGROSK2VFA5S3A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854819237 (2h37m24.31691172s) [noise=true , address=45.91.116.10:4082 , piece_id=YP64RUTIYQWN775JB527KG3KOJQMMIK6AJYRAJLPAIQBQH4NIZVA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854858622 (2h37m24.316951095s) [noise=true , address=45.91.116.12:4225 , piece_id=NF6W3OWZ7CD2XYLUPWAQX6EPGI753JCTD6GYGX6PSIIUQ5OD6SOA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.855251851 (2h37m24.317344324s) [noise=true , address=145.79.227.15:4876 , piece_id=Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.855651973 (2h37m24.317744456s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.856044019 (2h37m24.318136502s) [noise=true , address=145.79.227.19:4079 , piece_id=ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.856461354 (2h37m24.318553827s) [noise=true , address=145.79.227.14:4839 , piece_id=MPMNQZLCBFLMMBPYEQCJRJTTCZGZPZRGPDYW55PBM2I6V5RKXBAQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:40.657981922 (2h37m26.120074406s) [upload=2325]: Next returned nil segment Sep 23 16:08:40.658017650 (2h37m26.120110144s) [upload=2325]: Waiting for error group managing segments... Sep 23 16:08:40.965574730 (2h37m26.427667203s) [noise=true , address=45.91.116.5:4298 , piece_id=4T5H6GEBD5CWFHF7WKB4ISNKLF2ZS5DFVRZFWMQR7JUMWD6OFQBA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.017924353 (2h37m26.480016836s) [noise=true , address=45.91.116.9:4262 , piece_id=LO2THA7C52KXZGEPEFXDSJOUBQDUIAJCO6UNQF7ZXZ7ETHN4KIOA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.055061418 (2h37m26.517153891s) [noise=true , address=45.91.116.12:4225 , piece_id=NF6W3OWZ7CD2XYLUPWAQX6EPGI753JCTD6GYGX6PSIIUQ5OD6SOA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.059138873 (2h37m26.521231346s) [noise=true , address=77.90.31.10:4519 , piece_id=V7GQEGTIC643FJXEIXRRX64APJQD4YKJFLKEHXFBWXMRUGQ6I4KA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.068239702 (2h37m26.530332175s) [noise=true , address=45.91.116.11:4117 , piece_id=Q2P4XNRWOQEM257IMW253FIOTAWSPG3IEL55VTB2NHXSAQWAXH2A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.071035325 (2h37m26.533127798s) [noise=true , address=45.91.116.10:4082 , piece_id=YP64RUTIYQWN775JB527KG3KOJQMMIK6AJYRAJLPAIQBQH4NIZVA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.120390436 (2h37m26.582482919s) [noise=true , address=45.91.116.12:4227 , piece_id=IJTDTA3FDZCCVCCMWN3UWSNTWTSDU6UMY6F24GENRLVD3IXJ33WQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.262045750 (2h37m26.724138234s) [noise=true , address=77.90.31.12:4589 , piece_id=IRIEBUXYXWV2WHWNQDZT2ZPE75VHH4W3SURAYJ4MKMZVLIUUAUBQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:42.626787709 (2h37m28.088880182s) [noise=true , address=145.79.227.19:4079 , piece_id=ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:42.641124409 (2h37m28.103216882s) [noise=true , address=145.79.227.15:4876 , piece_id=Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.056725062 (2h37m28.518817545s) [noise=true , address=145.79.227.14:4839 , piece_id=MPMNQZLCBFLMMBPYEQCJRJTTCZGZPZRGPDYW55PBM2I6V5RKXBAQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.577366321 (2h37m29.039458804s) [noise=true , address=145.79.227.5:4222 , piece_id=HE27KIBGHAZPFT5FUX7FL4KLMWRG3YBZ74N4Q4FGROSK2VFA5S3A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.993428202 (2h37m29.455520675s) [noise=true , address=145.79.227.11:4731 , piece_id=46MQMGAXKX2MFI2IXJL7ZWTBVXYQRM6XA7XD565K2F73AKNSTN6A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.993452629 (2h37m29.455545102s) [seg_pos={0 0}, upload=2325]: Segment reached optimal threshold of 13 pieces. Sep 23 16:08:43.993624226 (2h37m29.455716709s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: expected piece hash; context canceled; EOF Sep 23 16:08:43.993637912 (2h37m29.455730395s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993678629 (2h37m29.455771112s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ): protocol: write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection Sep 23 16:08:43.993693137 (2h37m29.45578561s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993722223 (2h37m29.455814706s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Sep 23 16:08:43.993731370 (2h37m29.455823853s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993751087 (2h37m29.45584356s) [seg_pos={0 0}, upload=2325]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 16:08:43.993849845 (2h37m29.455942328s) [upload=2325]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 16:08:43.993858933 (2h37m29.455951416s) [upload=2325]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 16:08:44.305789411 (2h37m29.767881884s) [upload=2325]: Done uploading segments... Sep 23 16:08:44.305854725 (2h37m29.767947209s) [upload=2325]: Upload finished. err: <nil> Duration: 5.90815369s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.242566882s FAIL (2025-09-23 16:41:40 UTC) orbiter warning: context deadline exceeded Duration: 6.242614834s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.97554166s FAIL (2025-09-23 16:40:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA) context deadline exceeded; piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA) context deadline exceeded ======LOGS====== Sep 23 16:08:38.605894181 (2h37m24.067986664s) [upload=2325]: Starting upload Sep 23 16:08:38.606004200 (2h37m24.068096693s) [upload=2325]: Done starting upload Sep 23 16:08:38.606176218 (2h37m24.068268701s) [upload=2325]: Uploading segments... Sep 23 16:08:38.606277210 (2h37m24.068369683s) [upload=2325]: Got next segment. Inline: false Sep 23 16:08:38.606318859 (2h37m24.068411342s) [upload=2325]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 16:08:38.606324821 (2h37m24.068417304s) [upload=2325]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 16:08:38.852132441 (2h37m24.314224925s) [seg_pos={0 0}, upload=2325]: Begin upload segment... Sep 23 16:08:38.852225929 (2h37m24.314318402s) [seg_pos={0 0}, upload=2325]: Done begin upload segment. Sep 23 16:08:38.852268150 (2h37m24.314360623s) [noise=true , address=145.79.227.11:4731 , piece_id=46MQMGAXKX2MFI2IXJL7ZWTBVXYQRM6XA7XD565K2F73AKNSTN6A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.852333363 (2h37m24.314425866s) [noise=true , address=77.90.31.12:4589 , piece_id=IRIEBUXYXWV2WHWNQDZT2ZPE75VHH4W3SURAYJ4MKMZVLIUUAUBQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853073563 (2h37m24.315166037s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853408290 (2h37m24.315500763s) [noise=true , address=45.91.116.11:4117 , piece_id=Q2P4XNRWOQEM257IMW253FIOTAWSPG3IEL55VTB2NHXSAQWAXH2A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853673696 (2h37m24.315766179s) [noise=true , address=45.91.116.5:4298 , piece_id=4T5H6GEBD5CWFHF7WKB4ISNKLF2ZS5DFVRZFWMQR7JUMWD6OFQBA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853967866 (2h37m24.316060349s) [noise=true , address=45.91.116.12:4227 , piece_id=IJTDTA3FDZCCVCCMWN3UWSNTWTSDU6UMY6F24GENRLVD3IXJ33WQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854047137 (2h37m24.31613961s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854304707 (2h37m24.31639718s) [noise=true , address=45.91.116.9:4262 , piece_id=LO2THA7C52KXZGEPEFXDSJOUBQDUIAJCO6UNQF7ZXZ7ETHN4KIOA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854435016 (2h37m24.316527499s) [noise=true , address=77.90.31.10:4519 , piece_id=V7GQEGTIC643FJXEIXRRX64APJQD4YKJFLKEHXFBWXMRUGQ6I4KA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854553952 (2h37m24.316646425s) [noise=true , address=145.79.227.5:4222 , piece_id=HE27KIBGHAZPFT5FUX7FL4KLMWRG3YBZ74N4Q4FGROSK2VFA5S3A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854819237 (2h37m24.31691172s) [noise=true , address=45.91.116.10:4082 , piece_id=YP64RUTIYQWN775JB527KG3KOJQMMIK6AJYRAJLPAIQBQH4NIZVA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854858622 (2h37m24.316951095s) [noise=true , address=45.91.116.12:4225 , piece_id=NF6W3OWZ7CD2XYLUPWAQX6EPGI753JCTD6GYGX6PSIIUQ5OD6SOA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.855251851 (2h37m24.317344324s) [noise=true , address=145.79.227.15:4876 , piece_id=Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.855651973 (2h37m24.317744456s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.856044019 (2h37m24.318136502s) [noise=true , address=145.79.227.19:4079 , piece_id=ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.856461354 (2h37m24.318553827s) [noise=true , address=145.79.227.14:4839 , piece_id=MPMNQZLCBFLMMBPYEQCJRJTTCZGZPZRGPDYW55PBM2I6V5RKXBAQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:40.657981922 (2h37m26.120074406s) [upload=2325]: Next returned nil segment Sep 23 16:08:40.658017650 (2h37m26.120110144s) [upload=2325]: Waiting for error group managing segments... Sep 23 16:08:40.965574730 (2h37m26.427667203s) [noise=true , address=45.91.116.5:4298 , piece_id=4T5H6GEBD5CWFHF7WKB4ISNKLF2ZS5DFVRZFWMQR7JUMWD6OFQBA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.017924353 (2h37m26.480016836s) [noise=true , address=45.91.116.9:4262 , piece_id=LO2THA7C52KXZGEPEFXDSJOUBQDUIAJCO6UNQF7ZXZ7ETHN4KIOA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.055061418 (2h37m26.517153891s) [noise=true , address=45.91.116.12:4225 , piece_id=NF6W3OWZ7CD2XYLUPWAQX6EPGI753JCTD6GYGX6PSIIUQ5OD6SOA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.059138873 (2h37m26.521231346s) [noise=true , address=77.90.31.10:4519 , piece_id=V7GQEGTIC643FJXEIXRRX64APJQD4YKJFLKEHXFBWXMRUGQ6I4KA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.068239702 (2h37m26.530332175s) [noise=true , address=45.91.116.11:4117 , piece_id=Q2P4XNRWOQEM257IMW253FIOTAWSPG3IEL55VTB2NHXSAQWAXH2A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.071035325 (2h37m26.533127798s) [noise=true , address=45.91.116.10:4082 , piece_id=YP64RUTIYQWN775JB527KG3KOJQMMIK6AJYRAJLPAIQBQH4NIZVA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.120390436 (2h37m26.582482919s) [noise=true , address=45.91.116.12:4227 , piece_id=IJTDTA3FDZCCVCCMWN3UWSNTWTSDU6UMY6F24GENRLVD3IXJ33WQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.262045750 (2h37m26.724138234s) [noise=true , address=77.90.31.12:4589 , piece_id=IRIEBUXYXWV2WHWNQDZT2ZPE75VHH4W3SURAYJ4MKMZVLIUUAUBQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:42.626787709 (2h37m28.088880182s) [noise=true , address=145.79.227.19:4079 , piece_id=ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:42.641124409 (2h37m28.103216882s) [noise=true , address=145.79.227.15:4876 , piece_id=Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.056725062 (2h37m28.518817545s) [noise=true , address=145.79.227.14:4839 , piece_id=MPMNQZLCBFLMMBPYEQCJRJTTCZGZPZRGPDYW55PBM2I6V5RKXBAQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.577366321 (2h37m29.039458804s) [noise=true , address=145.79.227.5:4222 , piece_id=HE27KIBGHAZPFT5FUX7FL4KLMWRG3YBZ74N4Q4FGROSK2VFA5S3A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.993428202 (2h37m29.455520675s) [noise=true , address=145.79.227.11:4731 , piece_id=46MQMGAXKX2MFI2IXJL7ZWTBVXYQRM6XA7XD565K2F73AKNSTN6A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.993452629 (2h37m29.455545102s) [seg_pos={0 0}, upload=2325]: Segment reached optimal threshold of 13 pieces. Sep 23 16:08:43.993624226 (2h37m29.455716709s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: expected piece hash; context canceled; EOF Sep 23 16:08:43.993637912 (2h37m29.455730395s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993678629 (2h37m29.455771112s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ): protocol: write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection Sep 23 16:08:43.993693137 (2h37m29.45578561s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993722223 (2h37m29.455814706s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Sep 23 16:08:43.993731370 (2h37m29.455823853s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993751087 (2h37m29.45584356s) [seg_pos={0 0}, upload=2325]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 16:08:43.993849845 (2h37m29.455942328s) [upload=2325]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 16:08:43.993858933 (2h37m29.455951416s) [upload=2325]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 16:08:44.305789411 (2h37m29.767881884s) [upload=2325]: Done uploading segments... Sep 23 16:08:44.305854725 (2h37m29.767947209s) [upload=2325]: Upload finished. err: <nil> Duration: 5.975588098s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.001286121s FAIL (2025-09-23 16:40:48 UTC) orbiter warning: context deadline exceeded Duration: 6.001312581s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.131294348s FAIL (2025-09-23 16:40:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA) context deadline exceeded; piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA) context deadline exceeded ======LOGS====== Sep 23 16:08:38.605894181 (2h37m24.067986664s) [upload=2325]: Starting upload Sep 23 16:08:38.606004200 (2h37m24.068096693s) [upload=2325]: Done starting upload Sep 23 16:08:38.606176218 (2h37m24.068268701s) [upload=2325]: Uploading segments... Sep 23 16:08:38.606277210 (2h37m24.068369683s) [upload=2325]: Got next segment. Inline: false Sep 23 16:08:38.606318859 (2h37m24.068411342s) [upload=2325]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 16:08:38.606324821 (2h37m24.068417304s) [upload=2325]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 16:08:38.852132441 (2h37m24.314224925s) [seg_pos={0 0}, upload=2325]: Begin upload segment... Sep 23 16:08:38.852225929 (2h37m24.314318402s) [seg_pos={0 0}, upload=2325]: Done begin upload segment. Sep 23 16:08:38.852268150 (2h37m24.314360623s) [noise=true , address=145.79.227.11:4731 , piece_id=46MQMGAXKX2MFI2IXJL7ZWTBVXYQRM6XA7XD565K2F73AKNSTN6A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.852333363 (2h37m24.314425866s) [noise=true , address=77.90.31.12:4589 , piece_id=IRIEBUXYXWV2WHWNQDZT2ZPE75VHH4W3SURAYJ4MKMZVLIUUAUBQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853073563 (2h37m24.315166037s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853408290 (2h37m24.315500763s) [noise=true , address=45.91.116.11:4117 , piece_id=Q2P4XNRWOQEM257IMW253FIOTAWSPG3IEL55VTB2NHXSAQWAXH2A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853673696 (2h37m24.315766179s) [noise=true , address=45.91.116.5:4298 , piece_id=4T5H6GEBD5CWFHF7WKB4ISNKLF2ZS5DFVRZFWMQR7JUMWD6OFQBA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853967866 (2h37m24.316060349s) [noise=true , address=45.91.116.12:4227 , piece_id=IJTDTA3FDZCCVCCMWN3UWSNTWTSDU6UMY6F24GENRLVD3IXJ33WQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854047137 (2h37m24.31613961s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854304707 (2h37m24.31639718s) [noise=true , address=45.91.116.9:4262 , piece_id=LO2THA7C52KXZGEPEFXDSJOUBQDUIAJCO6UNQF7ZXZ7ETHN4KIOA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854435016 (2h37m24.316527499s) [noise=true , address=77.90.31.10:4519 , piece_id=V7GQEGTIC643FJXEIXRRX64APJQD4YKJFLKEHXFBWXMRUGQ6I4KA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854553952 (2h37m24.316646425s) [noise=true , address=145.79.227.5:4222 , piece_id=HE27KIBGHAZPFT5FUX7FL4KLMWRG3YBZ74N4Q4FGROSK2VFA5S3A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854819237 (2h37m24.31691172s) [noise=true , address=45.91.116.10:4082 , piece_id=YP64RUTIYQWN775JB527KG3KOJQMMIK6AJYRAJLPAIQBQH4NIZVA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854858622 (2h37m24.316951095s) [noise=true , address=45.91.116.12:4225 , piece_id=NF6W3OWZ7CD2XYLUPWAQX6EPGI753JCTD6GYGX6PSIIUQ5OD6SOA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.855251851 (2h37m24.317344324s) [noise=true , address=145.79.227.15:4876 , piece_id=Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.855651973 (2h37m24.317744456s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.856044019 (2h37m24.318136502s) [noise=true , address=145.79.227.19:4079 , piece_id=ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.856461354 (2h37m24.318553827s) [noise=true , address=145.79.227.14:4839 , piece_id=MPMNQZLCBFLMMBPYEQCJRJTTCZGZPZRGPDYW55PBM2I6V5RKXBAQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:40.657981922 (2h37m26.120074406s) [upload=2325]: Next returned nil segment Sep 23 16:08:40.658017650 (2h37m26.120110144s) [upload=2325]: Waiting for error group managing segments... Sep 23 16:08:40.965574730 (2h37m26.427667203s) [noise=true , address=45.91.116.5:4298 , piece_id=4T5H6GEBD5CWFHF7WKB4ISNKLF2ZS5DFVRZFWMQR7JUMWD6OFQBA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.017924353 (2h37m26.480016836s) [noise=true , address=45.91.116.9:4262 , piece_id=LO2THA7C52KXZGEPEFXDSJOUBQDUIAJCO6UNQF7ZXZ7ETHN4KIOA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.055061418 (2h37m26.517153891s) [noise=true , address=45.91.116.12:4225 , piece_id=NF6W3OWZ7CD2XYLUPWAQX6EPGI753JCTD6GYGX6PSIIUQ5OD6SOA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.059138873 (2h37m26.521231346s) [noise=true , address=77.90.31.10:4519 , piece_id=V7GQEGTIC643FJXEIXRRX64APJQD4YKJFLKEHXFBWXMRUGQ6I4KA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.068239702 (2h37m26.530332175s) [noise=true , address=45.91.116.11:4117 , piece_id=Q2P4XNRWOQEM257IMW253FIOTAWSPG3IEL55VTB2NHXSAQWAXH2A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.071035325 (2h37m26.533127798s) [noise=true , address=45.91.116.10:4082 , piece_id=YP64RUTIYQWN775JB527KG3KOJQMMIK6AJYRAJLPAIQBQH4NIZVA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.120390436 (2h37m26.582482919s) [noise=true , address=45.91.116.12:4227 , piece_id=IJTDTA3FDZCCVCCMWN3UWSNTWTSDU6UMY6F24GENRLVD3IXJ33WQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.262045750 (2h37m26.724138234s) [noise=true , address=77.90.31.12:4589 , piece_id=IRIEBUXYXWV2WHWNQDZT2ZPE75VHH4W3SURAYJ4MKMZVLIUUAUBQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:42.626787709 (2h37m28.088880182s) [noise=true , address=145.79.227.19:4079 , piece_id=ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:42.641124409 (2h37m28.103216882s) [noise=true , address=145.79.227.15:4876 , piece_id=Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.056725062 (2h37m28.518817545s) [noise=true , address=145.79.227.14:4839 , piece_id=MPMNQZLCBFLMMBPYEQCJRJTTCZGZPZRGPDYW55PBM2I6V5RKXBAQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.577366321 (2h37m29.039458804s) [noise=true , address=145.79.227.5:4222 , piece_id=HE27KIBGHAZPFT5FUX7FL4KLMWRG3YBZ74N4Q4FGROSK2VFA5S3A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.993428202 (2h37m29.455520675s) [noise=true , address=145.79.227.11:4731 , piece_id=46MQMGAXKX2MFI2IXJL7ZWTBVXYQRM6XA7XD565K2F73AKNSTN6A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.993452629 (2h37m29.455545102s) [seg_pos={0 0}, upload=2325]: Segment reached optimal threshold of 13 pieces. Sep 23 16:08:43.993624226 (2h37m29.455716709s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: expected piece hash; context canceled; EOF Sep 23 16:08:43.993637912 (2h37m29.455730395s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993678629 (2h37m29.455771112s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ): protocol: write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection Sep 23 16:08:43.993693137 (2h37m29.45578561s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993722223 (2h37m29.455814706s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Sep 23 16:08:43.993731370 (2h37m29.455823853s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993751087 (2h37m29.45584356s) [seg_pos={0 0}, upload=2325]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 16:08:43.993849845 (2h37m29.455942328s) [upload=2325]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 16:08:43.993858933 (2h37m29.455951416s) [upload=2325]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 16:08:44.305789411 (2h37m29.767881884s) [upload=2325]: Done uploading segments... Sep 23 16:08:44.305854725 (2h37m29.767947209s) [upload=2325]: Upload finished. err: <nil> Duration: 6.131549676s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.273923537s FAIL (2025-09-23 16:40:11 UTC) orbiter warning: context deadline exceeded Duration: 6.273954515s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.054139352s FAIL (2025-09-23 16:39:54 UTC) orbiter warning: context deadline exceeded Duration: 6.054167135s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.163085826s FAIL (2025-09-23 16:39:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA) context deadline exceeded; piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA) context deadline exceeded ======LOGS====== Sep 23 16:08:38.605894181 (2h37m24.067986664s) [upload=2325]: Starting upload Sep 23 16:08:38.606004200 (2h37m24.068096693s) [upload=2325]: Done starting upload Sep 23 16:08:38.606176218 (2h37m24.068268701s) [upload=2325]: Uploading segments... Sep 23 16:08:38.606277210 (2h37m24.068369683s) [upload=2325]: Got next segment. Inline: false Sep 23 16:08:38.606318859 (2h37m24.068411342s) [upload=2325]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 16:08:38.606324821 (2h37m24.068417304s) [upload=2325]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 16:08:38.852132441 (2h37m24.314224925s) [seg_pos={0 0}, upload=2325]: Begin upload segment... Sep 23 16:08:38.852225929 (2h37m24.314318402s) [seg_pos={0 0}, upload=2325]: Done begin upload segment. Sep 23 16:08:38.852268150 (2h37m24.314360623s) [noise=true , address=145.79.227.11:4731 , piece_id=46MQMGAXKX2MFI2IXJL7ZWTBVXYQRM6XA7XD565K2F73AKNSTN6A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.852333363 (2h37m24.314425866s) [noise=true , address=77.90.31.12:4589 , piece_id=IRIEBUXYXWV2WHWNQDZT2ZPE75VHH4W3SURAYJ4MKMZVLIUUAUBQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853073563 (2h37m24.315166037s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853408290 (2h37m24.315500763s) [noise=true , address=45.91.116.11:4117 , piece_id=Q2P4XNRWOQEM257IMW253FIOTAWSPG3IEL55VTB2NHXSAQWAXH2A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853673696 (2h37m24.315766179s) [noise=true , address=45.91.116.5:4298 , piece_id=4T5H6GEBD5CWFHF7WKB4ISNKLF2ZS5DFVRZFWMQR7JUMWD6OFQBA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853967866 (2h37m24.316060349s) [noise=true , address=45.91.116.12:4227 , piece_id=IJTDTA3FDZCCVCCMWN3UWSNTWTSDU6UMY6F24GENRLVD3IXJ33WQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854047137 (2h37m24.31613961s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854304707 (2h37m24.31639718s) [noise=true , address=45.91.116.9:4262 , piece_id=LO2THA7C52KXZGEPEFXDSJOUBQDUIAJCO6UNQF7ZXZ7ETHN4KIOA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854435016 (2h37m24.316527499s) [noise=true , address=77.90.31.10:4519 , piece_id=V7GQEGTIC643FJXEIXRRX64APJQD4YKJFLKEHXFBWXMRUGQ6I4KA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854553952 (2h37m24.316646425s) [noise=true , address=145.79.227.5:4222 , piece_id=HE27KIBGHAZPFT5FUX7FL4KLMWRG3YBZ74N4Q4FGROSK2VFA5S3A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854819237 (2h37m24.31691172s) [noise=true , address=45.91.116.10:4082 , piece_id=YP64RUTIYQWN775JB527KG3KOJQMMIK6AJYRAJLPAIQBQH4NIZVA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854858622 (2h37m24.316951095s) [noise=true , address=45.91.116.12:4225 , piece_id=NF6W3OWZ7CD2XYLUPWAQX6EPGI753JCTD6GYGX6PSIIUQ5OD6SOA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.855251851 (2h37m24.317344324s) [noise=true , address=145.79.227.15:4876 , piece_id=Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.855651973 (2h37m24.317744456s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.856044019 (2h37m24.318136502s) [noise=true , address=145.79.227.19:4079 , piece_id=ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.856461354 (2h37m24.318553827s) [noise=true , address=145.79.227.14:4839 , piece_id=MPMNQZLCBFLMMBPYEQCJRJTTCZGZPZRGPDYW55PBM2I6V5RKXBAQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:40.657981922 (2h37m26.120074406s) [upload=2325]: Next returned nil segment Sep 23 16:08:40.658017650 (2h37m26.120110144s) [upload=2325]: Waiting for error group managing segments... Sep 23 16:08:40.965574730 (2h37m26.427667203s) [noise=true , address=45.91.116.5:4298 , piece_id=4T5H6GEBD5CWFHF7WKB4ISNKLF2ZS5DFVRZFWMQR7JUMWD6OFQBA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.017924353 (2h37m26.480016836s) [noise=true , address=45.91.116.9:4262 , piece_id=LO2THA7C52KXZGEPEFXDSJOUBQDUIAJCO6UNQF7ZXZ7ETHN4KIOA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.055061418 (2h37m26.517153891s) [noise=true , address=45.91.116.12:4225 , piece_id=NF6W3OWZ7CD2XYLUPWAQX6EPGI753JCTD6GYGX6PSIIUQ5OD6SOA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.059138873 (2h37m26.521231346s) [noise=true , address=77.90.31.10:4519 , piece_id=V7GQEGTIC643FJXEIXRRX64APJQD4YKJFLKEHXFBWXMRUGQ6I4KA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.068239702 (2h37m26.530332175s) [noise=true , address=45.91.116.11:4117 , piece_id=Q2P4XNRWOQEM257IMW253FIOTAWSPG3IEL55VTB2NHXSAQWAXH2A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.071035325 (2h37m26.533127798s) [noise=true , address=45.91.116.10:4082 , piece_id=YP64RUTIYQWN775JB527KG3KOJQMMIK6AJYRAJLPAIQBQH4NIZVA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.120390436 (2h37m26.582482919s) [noise=true , address=45.91.116.12:4227 , piece_id=IJTDTA3FDZCCVCCMWN3UWSNTWTSDU6UMY6F24GENRLVD3IXJ33WQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.262045750 (2h37m26.724138234s) [noise=true , address=77.90.31.12:4589 , piece_id=IRIEBUXYXWV2WHWNQDZT2ZPE75VHH4W3SURAYJ4MKMZVLIUUAUBQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:42.626787709 (2h37m28.088880182s) [noise=true , address=145.79.227.19:4079 , piece_id=ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:42.641124409 (2h37m28.103216882s) [noise=true , address=145.79.227.15:4876 , piece_id=Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.056725062 (2h37m28.518817545s) [noise=true , address=145.79.227.14:4839 , piece_id=MPMNQZLCBFLMMBPYEQCJRJTTCZGZPZRGPDYW55PBM2I6V5RKXBAQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.577366321 (2h37m29.039458804s) [noise=true , address=145.79.227.5:4222 , piece_id=HE27KIBGHAZPFT5FUX7FL4KLMWRG3YBZ74N4Q4FGROSK2VFA5S3A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.993428202 (2h37m29.455520675s) [noise=true , address=145.79.227.11:4731 , piece_id=46MQMGAXKX2MFI2IXJL7ZWTBVXYQRM6XA7XD565K2F73AKNSTN6A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.993452629 (2h37m29.455545102s) [seg_pos={0 0}, upload=2325]: Segment reached optimal threshold of 13 pieces. Sep 23 16:08:43.993624226 (2h37m29.455716709s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: expected piece hash; context canceled; EOF Sep 23 16:08:43.993637912 (2h37m29.455730395s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993678629 (2h37m29.455771112s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ): protocol: write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection Sep 23 16:08:43.993693137 (2h37m29.45578561s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993722223 (2h37m29.455814706s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Sep 23 16:08:43.993731370 (2h37m29.455823853s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993751087 (2h37m29.45584356s) [seg_pos={0 0}, upload=2325]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 16:08:43.993849845 (2h37m29.455942328s) [upload=2325]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 16:08:43.993858933 (2h37m29.455951416s) [upload=2325]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 16:08:44.305789411 (2h37m29.767881884s) [upload=2325]: Done uploading segments... Sep 23 16:08:44.305854725 (2h37m29.767947209s) [upload=2325]: Upload finished. err: <nil> Duration: 6.16311993s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.985575765s FAIL (2025-09-23 16:39:12 UTC) orbiter warning: context deadline exceeded Duration: 5.985601205s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.12892114s FAIL (2025-09-23 16:38:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA) context deadline exceeded; piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA) context deadline exceeded ======LOGS====== Sep 23 16:08:38.605894181 (2h37m24.067986664s) [upload=2325]: Starting upload Sep 23 16:08:38.606004200 (2h37m24.068096693s) [upload=2325]: Done starting upload Sep 23 16:08:38.606176218 (2h37m24.068268701s) [upload=2325]: Uploading segments... Sep 23 16:08:38.606277210 (2h37m24.068369683s) [upload=2325]: Got next segment. Inline: false Sep 23 16:08:38.606318859 (2h37m24.068411342s) [upload=2325]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 16:08:38.606324821 (2h37m24.068417304s) [upload=2325]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 16:08:38.852132441 (2h37m24.314224925s) [seg_pos={0 0}, upload=2325]: Begin upload segment... Sep 23 16:08:38.852225929 (2h37m24.314318402s) [seg_pos={0 0}, upload=2325]: Done begin upload segment. Sep 23 16:08:38.852268150 (2h37m24.314360623s) [noise=true , address=145.79.227.11:4731 , piece_id=46MQMGAXKX2MFI2IXJL7ZWTBVXYQRM6XA7XD565K2F73AKNSTN6A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.852333363 (2h37m24.314425866s) [noise=true , address=77.90.31.12:4589 , piece_id=IRIEBUXYXWV2WHWNQDZT2ZPE75VHH4W3SURAYJ4MKMZVLIUUAUBQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853073563 (2h37m24.315166037s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853408290 (2h37m24.315500763s) [noise=true , address=45.91.116.11:4117 , piece_id=Q2P4XNRWOQEM257IMW253FIOTAWSPG3IEL55VTB2NHXSAQWAXH2A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853673696 (2h37m24.315766179s) [noise=true , address=45.91.116.5:4298 , piece_id=4T5H6GEBD5CWFHF7WKB4ISNKLF2ZS5DFVRZFWMQR7JUMWD6OFQBA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.853967866 (2h37m24.316060349s) [noise=true , address=45.91.116.12:4227 , piece_id=IJTDTA3FDZCCVCCMWN3UWSNTWTSDU6UMY6F24GENRLVD3IXJ33WQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854047137 (2h37m24.31613961s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854304707 (2h37m24.31639718s) [noise=true , address=45.91.116.9:4262 , piece_id=LO2THA7C52KXZGEPEFXDSJOUBQDUIAJCO6UNQF7ZXZ7ETHN4KIOA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854435016 (2h37m24.316527499s) [noise=true , address=77.90.31.10:4519 , piece_id=V7GQEGTIC643FJXEIXRRX64APJQD4YKJFLKEHXFBWXMRUGQ6I4KA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854553952 (2h37m24.316646425s) [noise=true , address=145.79.227.5:4222 , piece_id=HE27KIBGHAZPFT5FUX7FL4KLMWRG3YBZ74N4Q4FGROSK2VFA5S3A, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854819237 (2h37m24.31691172s) [noise=true , address=45.91.116.10:4082 , piece_id=YP64RUTIYQWN775JB527KG3KOJQMMIK6AJYRAJLPAIQBQH4NIZVA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.854858622 (2h37m24.316951095s) [noise=true , address=45.91.116.12:4225 , piece_id=NF6W3OWZ7CD2XYLUPWAQX6EPGI753JCTD6GYGX6PSIIUQ5OD6SOA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.855251851 (2h37m24.317344324s) [noise=true , address=145.79.227.15:4876 , piece_id=Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.855651973 (2h37m24.317744456s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.856044019 (2h37m24.318136502s) [noise=true , address=145.79.227.19:4079 , piece_id=ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:38.856461354 (2h37m24.318553827s) [noise=true , address=145.79.227.14:4839 , piece_id=MPMNQZLCBFLMMBPYEQCJRJTTCZGZPZRGPDYW55PBM2I6V5RKXBAQ, seg_pos={0 0}, upload=2325]: Uploading piece... Sep 23 16:08:40.657981922 (2h37m26.120074406s) [upload=2325]: Next returned nil segment Sep 23 16:08:40.658017650 (2h37m26.120110144s) [upload=2325]: Waiting for error group managing segments... Sep 23 16:08:40.965574730 (2h37m26.427667203s) [noise=true , address=45.91.116.5:4298 , piece_id=4T5H6GEBD5CWFHF7WKB4ISNKLF2ZS5DFVRZFWMQR7JUMWD6OFQBA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.017924353 (2h37m26.480016836s) [noise=true , address=45.91.116.9:4262 , piece_id=LO2THA7C52KXZGEPEFXDSJOUBQDUIAJCO6UNQF7ZXZ7ETHN4KIOA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.055061418 (2h37m26.517153891s) [noise=true , address=45.91.116.12:4225 , piece_id=NF6W3OWZ7CD2XYLUPWAQX6EPGI753JCTD6GYGX6PSIIUQ5OD6SOA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.059138873 (2h37m26.521231346s) [noise=true , address=77.90.31.10:4519 , piece_id=V7GQEGTIC643FJXEIXRRX64APJQD4YKJFLKEHXFBWXMRUGQ6I4KA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.068239702 (2h37m26.530332175s) [noise=true , address=45.91.116.11:4117 , piece_id=Q2P4XNRWOQEM257IMW253FIOTAWSPG3IEL55VTB2NHXSAQWAXH2A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.071035325 (2h37m26.533127798s) [noise=true , address=45.91.116.10:4082 , piece_id=YP64RUTIYQWN775JB527KG3KOJQMMIK6AJYRAJLPAIQBQH4NIZVA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.120390436 (2h37m26.582482919s) [noise=true , address=45.91.116.12:4227 , piece_id=IJTDTA3FDZCCVCCMWN3UWSNTWTSDU6UMY6F24GENRLVD3IXJ33WQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:41.262045750 (2h37m26.724138234s) [noise=true , address=77.90.31.12:4589 , piece_id=IRIEBUXYXWV2WHWNQDZT2ZPE75VHH4W3SURAYJ4MKMZVLIUUAUBQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:42.626787709 (2h37m28.088880182s) [noise=true , address=145.79.227.19:4079 , piece_id=ETJUASKFTZZHS7LTVHCWQDCG6T4VKEQOAUZO6G422UQFFYQRBCTA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:42.641124409 (2h37m28.103216882s) [noise=true , address=145.79.227.15:4876 , piece_id=Q6S22GHTR2N3NF7X63GZ56XTFC52CSYI7QED2XDQYQA3BFJLFWAA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.056725062 (2h37m28.518817545s) [noise=true , address=145.79.227.14:4839 , piece_id=MPMNQZLCBFLMMBPYEQCJRJTTCZGZPZRGPDYW55PBM2I6V5RKXBAQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.577366321 (2h37m29.039458804s) [noise=true , address=145.79.227.5:4222 , piece_id=HE27KIBGHAZPFT5FUX7FL4KLMWRG3YBZ74N4Q4FGROSK2VFA5S3A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.993428202 (2h37m29.455520675s) [noise=true , address=145.79.227.11:4731 , piece_id=46MQMGAXKX2MFI2IXJL7ZWTBVXYQRM6XA7XD565K2F73AKNSTN6A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Sep 23 16:08:43.993452629 (2h37m29.455545102s) [seg_pos={0 0}, upload=2325]: Segment reached optimal threshold of 13 pieces. Sep 23 16:08:43.993624226 (2h37m29.455716709s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: expected piece hash; context canceled; EOF Sep 23 16:08:43.993637912 (2h37m29.455730395s) [noise=true , address=145.79.227.6:4259 , piece_id=M3MEYQ2XOMDXINQT2MKK2PPQ5DLFLBJMHVI4PWIRXLKSJK74BDMA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993678629 (2h37m29.455771112s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ): protocol: write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52350->145.79.227.13:4801: use of closed network connection Sep 23 16:08:43.993693137 (2h37m29.45578561s) [noise=true , address=145.79.227.13:4801 , piece_id=Z3JOWJVUQ5BCG36PVVCXWCULB5PDWR4HJWN7NPQF4VDENEMHPCEA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993722223 (2h37m29.455814706s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Sep 23 16:08:43.993731370 (2h37m29.455823853s) [noise=true , address=145.79.227.3:4150 , piece_id=A6PSDWLOA3E2OGLJ3PW62BULLD2HAGWALQKCMNAGDURXSB3EUOQA, seg_pos={0 0}, upload=2325]: Long tail cancellation due to optimal threshold Sep 23 16:08:43.993751087 (2h37m29.45584356s) [seg_pos={0 0}, upload=2325]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 16:08:43.993849845 (2h37m29.455942328s) [upload=2325]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 16:08:43.993858933 (2h37m29.455951416s) [upload=2325]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 16:08:44.305789411 (2h37m29.767881884s) [upload=2325]: Done uploading segments... Sep 23 16:08:44.305854725 (2h37m29.767947209s) [upload=2325]: Upload finished. err: <nil> Duration: 6.129027102s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)