Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.25550606s trace: https://jaeger.dev.storj.io/trace/d53ad46a3bad038 FAIL (2025-11-13 14:51:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ... Nov 13 14:17:39.718806610 (102h13m38.270277071s) [noise=true , address=5.231.73.3:4030 , piece_id=O4B5VTAWE2WNRG2GZT4OYOEGZLDNYDTHZOT3WETYA3P4XHQR5AOQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.718869189 (102h13m38.27033964s) [noise=true , address=45.91.116.7:4016 , piece_id=BXDTJNCLTSVXQUCC4RO2C2JHZ26XHIJPTHITUO7TXQY6CAY6AC2A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719243782 (102h13m38.270714243s) [noise=true , address=145.79.227.38:4984 , piece_id=LP425WCAJOPGTKLRDOU6V2M3LWRGZ3DZLF4JL4L5QDHW4V37TOZA, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719600372 (102h13m38.271070833s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719757630 (102h13m38.271228102s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719982179 (102h13m38.27145265s) [noise=true , address=5.231.73.2:4018 , piece_id=KP2VQBGZYKUDBJNAIKXV6VRHPK6V3UI6X3SD6HIJOHDTZ6Z6EGLQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720312697 (102h13m38.271783169s) [noise=true , address=77.90.31.17:4776 , piece_id=254SH54AOFXNCTTM6QF3AUN7DYYRE2EKBHDBEUKLHEJMKDBXSVDQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720732096 (102h13m38.272202567s) [noise=true , address=45.91.116.6:4993 , piece_id=B7J3JA5UFDKUW6Q6YACXNQY7EUH7PXYRFUOMMLQKOCWJ6NZ3G3DQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720836485 (102h13m38.272306946s) [noise=true , address=45.91.116.11:4145 , piece_id=7SLQGSEAL7WCGUQ6TRYWNM6F57TZKH6PNDNVXWQGTZ6LKKVXBONA, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.721147126 (102h13m38.272617587s) [noise=true , address=77.90.31.14:4678 , piece_id=D4HEZCFBVRL5CMXNWRCWXFOA3UMT2NXCMIA2L4ACWQ25O3PU4B4A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.721537239 (102h13m38.2730077s) [noise=true , address=5.231.73.4:4022 , piece_id=GABEYHU5QIUMMQYAHKVXLY6QFY4AZRXMVPSDJ3EO6WOIKA7E4U4Q, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.722351200 (102h13m38.273821661s) [noise=true , address=45.91.116.14:4203 , piece_id=SU2IO7NSAAQP4ZEY4QIHXQYOHFFDOSGFQEHFJG44QLIPA37DUA6Q, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.723267113 (102h13m38.274737584s) [noise=true , address=5.231.73.5:4019 , piece_id=H6CZGSTCCYMOZVTSJ2Y5XAKGZA26EQ7E2NONL6DWZGC7Q2LORHBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.723886824 (102h13m38.275357285s) [noise=true , address=64.49.13.26:4948 , piece_id=MQ6RQQQ36YHFSTJYDE44AATC7LQPIFCB7QOS72DMTWRETZVAQWDQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.724315690 (102h13m38.275786151s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.724792528 (102h13m38.276262989s) [noise=true , address=64.49.13.138:4492 , piece_id=CXNCL6OGRP6C4BXBSH7URFSYJELZLM62KVHGCDJQPD7FVTB67Y3A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.725327326 (102h13m38.276797787s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:41.027262394 (102h13m39.578732845s) [upload=89995]: Next returned nil segment Nov 13 14:17:41.027282332 (102h13m39.578752783s) [upload=89995]: Waiting for error group managing segments... Nov 13 14:17:41.357905237 (102h13m39.909375698s) [noise=true , address=64.49.13.26:4948 , piece_id=MQ6RQQQ36YHFSTJYDE44AATC7LQPIFCB7QOS72DMTWRETZVAQWDQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.433145731 (102h13m39.984616192s) [noise=true , address=64.49.13.154:4929 , piece_id=2K55IIOVKBDVMEO3QKXXOZ65X35SQIVSEN6RJAW6OWEGE6ZKYCRA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.460916682 (102h13m40.012387133s) [noise=true , address=5.231.73.2:4018 , piece_id=KP2VQBGZYKUDBJNAIKXV6VRHPK6V3UI6X3SD6HIJOHDTZ6Z6EGLQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.497737346 (102h13m40.049207797s) [noise=true , address=5.231.73.5:4019 , piece_id=H6CZGSTCCYMOZVTSJ2Y5XAKGZA26EQ7E2NONL6DWZGC7Q2LORHBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.511205242 (102h13m40.062675703s) [noise=true , address=5.231.73.3:4030 , piece_id=O4B5VTAWE2WNRG2GZT4OYOEGZLDNYDTHZOT3WETYA3P4XHQR5AOQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.536492093 (102h13m40.087962565s) [noise=true , address=5.231.73.4:4022 , piece_id=GABEYHU5QIUMMQYAHKVXLY6QFY4AZRXMVPSDJ3EO6WOIKA7E4U4Q, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.553999952 (102h13m40.105470433s) [noise=true , address=64.49.13.138:4492 , piece_id=CXNCL6OGRP6C4BXBSH7URFSYJELZLM62KVHGCDJQPD7FVTB67Y3A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.668337736 (102h13m40.219808187s) [noise=true , address=45.91.116.7:4016 , piece_id=BXDTJNCLTSVXQUCC4RO2C2JHZ26XHIJPTHITUO7TXQY6CAY6AC2A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.674376285 (102h13m40.225846746s) [noise=true , address=77.90.31.17:4776 , piece_id=254SH54AOFXNCTTM6QF3AUN7DYYRE2EKBHDBEUKLHEJMKDBXSVDQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.678052054 (102h13m40.229522505s) [noise=true , address=145.79.227.38:4984 , piece_id=LP425WCAJOPGTKLRDOU6V2M3LWRGZ3DZLF4JL4L5QDHW4V37TOZA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.761077612 (102h13m40.312548063s) [noise=true , address=77.90.31.14:4678 , piece_id=D4HEZCFBVRL5CMXNWRCWXFOA3UMT2NXCMIA2L4ACWQ25O3PU4B4A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.771558030 (102h13m40.323028481s) [noise=true , address=45.91.116.6:4993 , piece_id=B7J3JA5UFDKUW6Q6YACXNQY7EUH7PXYRFUOMMLQKOCWJ6NZ3G3DQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.772042953 (102h13m40.323513414s) [noise=true , address=45.91.116.11:4145 , piece_id=7SLQGSEAL7WCGUQ6TRYWNM6F57TZKH6PNDNVXWQGTZ6LKKVXBONA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.775497421 (102h13m40.326967872s) [noise=true , address=45.91.116.14:4203 , piece_id=SU2IO7NSAAQP4ZEY4QIHXQYOHFFDOSGFQEHFJG44QLIPA37DUA6Q, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.775520495 (102h13m40.326990956s) [seg_pos={0 0}, upload=89995]: Segment reached optimal threshold of 14 pieces. Nov 13 14:17:41.775700267 (102h13m40.327170729s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.775707361 (102h13m40.327177812s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.775974550 (102h13m40.327445001s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.775982485 (102h13m40.327452936s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776090991 (102h13m40.327561442s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.776096271 (102h13m40.327566732s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776208986 (102h13m40.327679437s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.776214626 (102h13m40.327685087s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776231869 (102h13m40.32770232s) [seg_pos={0 0}, upload=89995]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 13 14:17:41.776317181 (102h13m40.327787632s) [upload=89995]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:17:41.776322282 (102h13m40.327792743s) [upload=89995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:18:48.053973011 (102h14m46.605443472s) [upload=89995]: Done uploading segments... Nov 13 14:18:48.054067461 (102h14m46.605537912s) [upload=89995]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.255520809s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.222663367s trace: https://jaeger.dev.storj.io/trace/d53ad46a3bad038 FAIL (2025-11-13 14:51:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ... Nov 13 14:17:39.718806610 (102h13m38.270277071s) [noise=true , address=5.231.73.3:4030 , piece_id=O4B5VTAWE2WNRG2GZT4OYOEGZLDNYDTHZOT3WETYA3P4XHQR5AOQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.718869189 (102h13m38.27033964s) [noise=true , address=45.91.116.7:4016 , piece_id=BXDTJNCLTSVXQUCC4RO2C2JHZ26XHIJPTHITUO7TXQY6CAY6AC2A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719243782 (102h13m38.270714243s) [noise=true , address=145.79.227.38:4984 , piece_id=LP425WCAJOPGTKLRDOU6V2M3LWRGZ3DZLF4JL4L5QDHW4V37TOZA, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719600372 (102h13m38.271070833s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719757630 (102h13m38.271228102s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719982179 (102h13m38.27145265s) [noise=true , address=5.231.73.2:4018 , piece_id=KP2VQBGZYKUDBJNAIKXV6VRHPK6V3UI6X3SD6HIJOHDTZ6Z6EGLQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720312697 (102h13m38.271783169s) [noise=true , address=77.90.31.17:4776 , piece_id=254SH54AOFXNCTTM6QF3AUN7DYYRE2EKBHDBEUKLHEJMKDBXSVDQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720732096 (102h13m38.272202567s) [noise=true , address=45.91.116.6:4993 , piece_id=B7J3JA5UFDKUW6Q6YACXNQY7EUH7PXYRFUOMMLQKOCWJ6NZ3G3DQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720836485 (102h13m38.272306946s) [noise=true , address=45.91.116.11:4145 , piece_id=7SLQGSEAL7WCGUQ6TRYWNM6F57TZKH6PNDNVXWQGTZ6LKKVXBONA, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.721147126 (102h13m38.272617587s) [noise=true , address=77.90.31.14:4678 , piece_id=D4HEZCFBVRL5CMXNWRCWXFOA3UMT2NXCMIA2L4ACWQ25O3PU4B4A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.721537239 (102h13m38.2730077s) [noise=true , address=5.231.73.4:4022 , piece_id=GABEYHU5QIUMMQYAHKVXLY6QFY4AZRXMVPSDJ3EO6WOIKA7E4U4Q, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.722351200 (102h13m38.273821661s) [noise=true , address=45.91.116.14:4203 , piece_id=SU2IO7NSAAQP4ZEY4QIHXQYOHFFDOSGFQEHFJG44QLIPA37DUA6Q, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.723267113 (102h13m38.274737584s) [noise=true , address=5.231.73.5:4019 , piece_id=H6CZGSTCCYMOZVTSJ2Y5XAKGZA26EQ7E2NONL6DWZGC7Q2LORHBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.723886824 (102h13m38.275357285s) [noise=true , address=64.49.13.26:4948 , piece_id=MQ6RQQQ36YHFSTJYDE44AATC7LQPIFCB7QOS72DMTWRETZVAQWDQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.724315690 (102h13m38.275786151s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.724792528 (102h13m38.276262989s) [noise=true , address=64.49.13.138:4492 , piece_id=CXNCL6OGRP6C4BXBSH7URFSYJELZLM62KVHGCDJQPD7FVTB67Y3A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.725327326 (102h13m38.276797787s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:41.027262394 (102h13m39.578732845s) [upload=89995]: Next returned nil segment Nov 13 14:17:41.027282332 (102h13m39.578752783s) [upload=89995]: Waiting for error group managing segments... Nov 13 14:17:41.357905237 (102h13m39.909375698s) [noise=true , address=64.49.13.26:4948 , piece_id=MQ6RQQQ36YHFSTJYDE44AATC7LQPIFCB7QOS72DMTWRETZVAQWDQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.433145731 (102h13m39.984616192s) [noise=true , address=64.49.13.154:4929 , piece_id=2K55IIOVKBDVMEO3QKXXOZ65X35SQIVSEN6RJAW6OWEGE6ZKYCRA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.460916682 (102h13m40.012387133s) [noise=true , address=5.231.73.2:4018 , piece_id=KP2VQBGZYKUDBJNAIKXV6VRHPK6V3UI6X3SD6HIJOHDTZ6Z6EGLQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.497737346 (102h13m40.049207797s) [noise=true , address=5.231.73.5:4019 , piece_id=H6CZGSTCCYMOZVTSJ2Y5XAKGZA26EQ7E2NONL6DWZGC7Q2LORHBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.511205242 (102h13m40.062675703s) [noise=true , address=5.231.73.3:4030 , piece_id=O4B5VTAWE2WNRG2GZT4OYOEGZLDNYDTHZOT3WETYA3P4XHQR5AOQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.536492093 (102h13m40.087962565s) [noise=true , address=5.231.73.4:4022 , piece_id=GABEYHU5QIUMMQYAHKVXLY6QFY4AZRXMVPSDJ3EO6WOIKA7E4U4Q, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.553999952 (102h13m40.105470433s) [noise=true , address=64.49.13.138:4492 , piece_id=CXNCL6OGRP6C4BXBSH7URFSYJELZLM62KVHGCDJQPD7FVTB67Y3A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.668337736 (102h13m40.219808187s) [noise=true , address=45.91.116.7:4016 , piece_id=BXDTJNCLTSVXQUCC4RO2C2JHZ26XHIJPTHITUO7TXQY6CAY6AC2A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.674376285 (102h13m40.225846746s) [noise=true , address=77.90.31.17:4776 , piece_id=254SH54AOFXNCTTM6QF3AUN7DYYRE2EKBHDBEUKLHEJMKDBXSVDQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.678052054 (102h13m40.229522505s) [noise=true , address=145.79.227.38:4984 , piece_id=LP425WCAJOPGTKLRDOU6V2M3LWRGZ3DZLF4JL4L5QDHW4V37TOZA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.761077612 (102h13m40.312548063s) [noise=true , address=77.90.31.14:4678 , piece_id=D4HEZCFBVRL5CMXNWRCWXFOA3UMT2NXCMIA2L4ACWQ25O3PU4B4A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.771558030 (102h13m40.323028481s) [noise=true , address=45.91.116.6:4993 , piece_id=B7J3JA5UFDKUW6Q6YACXNQY7EUH7PXYRFUOMMLQKOCWJ6NZ3G3DQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.772042953 (102h13m40.323513414s) [noise=true , address=45.91.116.11:4145 , piece_id=7SLQGSEAL7WCGUQ6TRYWNM6F57TZKH6PNDNVXWQGTZ6LKKVXBONA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.775497421 (102h13m40.326967872s) [noise=true , address=45.91.116.14:4203 , piece_id=SU2IO7NSAAQP4ZEY4QIHXQYOHFFDOSGFQEHFJG44QLIPA37DUA6Q, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.775520495 (102h13m40.326990956s) [seg_pos={0 0}, upload=89995]: Segment reached optimal threshold of 14 pieces. Nov 13 14:17:41.775700267 (102h13m40.327170729s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.775707361 (102h13m40.327177812s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.775974550 (102h13m40.327445001s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.775982485 (102h13m40.327452936s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776090991 (102h13m40.327561442s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.776096271 (102h13m40.327566732s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776208986 (102h13m40.327679437s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.776214626 (102h13m40.327685087s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776231869 (102h13m40.32770232s) [seg_pos={0 0}, upload=89995]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 13 14:17:41.776317181 (102h13m40.327787632s) [upload=89995]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:17:41.776322282 (102h13m40.327792743s) [upload=89995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:18:48.053973011 (102h14m46.605443472s) [upload=89995]: Done uploading segments... Nov 13 14:18:48.054067461 (102h14m46.605537912s) [upload=89995]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.222695129s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 7.192865777s trace: https://jaeger.dev.storj.io/trace/403ba97a5a1cde7d FAIL (2025-11-13 14:51:17 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:00.282283372 (102h11m58.833753833s) [upload=89970]: Starting upload Nov 13 14:16:00.282383773 (102h11m58.833854234s) [upload=89970]: Done starting upload Nov 13 14:16:00.282466601 (102h11m58.833937052s) [upload=89970]: Uploading segments... Nov 13 14:16:00.282681841 (102h11m58.834152302s) [upload=89970]: Got next segment. Inline: false Nov 13 14:16:00.282731475 (102h11m58.834201936s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:00.282737627 (102h11m58.834208088s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 14:16:02.313071205 (102h12m0.864541666s) [seg_pos={0 0}, upload=89970]: Begin upload segment... Nov 13 14:16:02.313149905 (102h12m0.864620376s) [seg_pos={0 0}, upload=89970]: Done begin upload segment. Nov 13 14:16:02.313271697 (102h12m0.864742158s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.313324648 (102h12m0.864795129s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314320904 (102h12m0.865791355s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314744151 (102h12m0.866214642s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315158950 (102h12m0.866629421s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315599659 (102h12m0.867070121s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315928506 (102h12m0.867398967s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.316477040 (102h12m0.867947501s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317049180 (102h12m0.868519651s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317427380 (102h12m0.868897841s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317934746 (102h12m0.869405207s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318376678 (102h12m0.869847139s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318812577 (102h12m0.870283038s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319367303 (102h12m0.870837765s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319957317 (102h12m0.871427768s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.320489491 (102h12m0.871959952s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:03.991369116 (102h12m2.542839567s) [upload=89970]: Next returned nil segment Nov 13 14:16:03.991386198 (102h12m2.542856649s) [upload=89970]: Waiting for error group managing segments... Nov 13 14:16:04.278532114 (102h12m2.830002575s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.279571643 (102h12m2.831042105s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.289684120 (102h12m2.841154581s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.300556964 (102h12m2.852027435s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.301333473 (102h12m2.852803934s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302408349 (102h12m2.853878821s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302631235 (102h12m2.854101716s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.335556962 (102h12m2.887027413s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.346910641 (102h12m2.898381102s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.371810816 (102h12m2.923281277s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377236428 (102h12m2.928706889s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377262828 (102h12m2.928733289s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383693804 (102h12m2.935164265s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383710837 (102h12m2.935181288s) [seg_pos={0 0}, upload=89970]: Segment reached optimal threshold of 13 pieces. Nov 13 14:16:04.383822670 (102h12m2.935293121s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383922450 (102h12m2.935392911s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ): protocol: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection Nov 13 14:16:04.383929954 (102h12m2.935400415s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384043279 (102h12m2.93551374s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection Nov 13 14:16:04.384050112 (102h12m2.935520563s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384066143 (102h12m2.935536604s) [seg_pos={0 0}, upload=89970]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Nov 13 14:16:04.384159561 (102h12m2.935630013s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:16:04.384171734 (102h12m2.935642195s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:16:04.934815134 (102h12m3.486285595s) [upload=89970]: Done uploading segments... Nov 13 14:16:04.934864327 (102h12m3.486334788s) [upload=89970]: Upload finished. err: <nil> Duration: 7.192926422s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.095395506s trace: https://jaeger.dev.storj.io/trace/afd242eeed2927 FAIL (2025-11-13 14:51:07 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.095441903s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.137654586s trace: https://jaeger.dev.storj.io/trace/403ba97a5a1cde7d FAIL (2025-11-13 14:51:01 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:00.282283372 (102h11m58.833753833s) [upload=89970]: Starting upload Nov 13 14:16:00.282383773 (102h11m58.833854234s) [upload=89970]: Done starting upload Nov 13 14:16:00.282466601 (102h11m58.833937052s) [upload=89970]: Uploading segments... Nov 13 14:16:00.282681841 (102h11m58.834152302s) [upload=89970]: Got next segment. Inline: false Nov 13 14:16:00.282731475 (102h11m58.834201936s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:00.282737627 (102h11m58.834208088s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 14:16:02.313071205 (102h12m0.864541666s) [seg_pos={0 0}, upload=89970]: Begin upload segment... Nov 13 14:16:02.313149905 (102h12m0.864620376s) [seg_pos={0 0}, upload=89970]: Done begin upload segment. Nov 13 14:16:02.313271697 (102h12m0.864742158s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.313324648 (102h12m0.864795129s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314320904 (102h12m0.865791355s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314744151 (102h12m0.866214642s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315158950 (102h12m0.866629421s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315599659 (102h12m0.867070121s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315928506 (102h12m0.867398967s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.316477040 (102h12m0.867947501s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317049180 (102h12m0.868519651s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317427380 (102h12m0.868897841s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317934746 (102h12m0.869405207s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318376678 (102h12m0.869847139s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318812577 (102h12m0.870283038s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319367303 (102h12m0.870837765s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319957317 (102h12m0.871427768s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.320489491 (102h12m0.871959952s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:03.991369116 (102h12m2.542839567s) [upload=89970]: Next returned nil segment Nov 13 14:16:03.991386198 (102h12m2.542856649s) [upload=89970]: Waiting for error group managing segments... Nov 13 14:16:04.278532114 (102h12m2.830002575s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.279571643 (102h12m2.831042105s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.289684120 (102h12m2.841154581s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.300556964 (102h12m2.852027435s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.301333473 (102h12m2.852803934s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302408349 (102h12m2.853878821s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302631235 (102h12m2.854101716s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.335556962 (102h12m2.887027413s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.346910641 (102h12m2.898381102s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.371810816 (102h12m2.923281277s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377236428 (102h12m2.928706889s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377262828 (102h12m2.928733289s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383693804 (102h12m2.935164265s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383710837 (102h12m2.935181288s) [seg_pos={0 0}, upload=89970]: Segment reached optimal threshold of 13 pieces. Nov 13 14:16:04.383822670 (102h12m2.935293121s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383922450 (102h12m2.935392911s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ): protocol: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection Nov 13 14:16:04.383929954 (102h12m2.935400415s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384043279 (102h12m2.93551374s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection Nov 13 14:16:04.384050112 (102h12m2.935520563s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384066143 (102h12m2.935536604s) [seg_pos={0 0}, upload=89970]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Nov 13 14:16:04.384159561 (102h12m2.935630013s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:16:04.384171734 (102h12m2.935642195s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:16:04.934815134 (102h12m3.486285595s) [upload=89970]: Done uploading segments... Nov 13 14:16:04.934864327 (102h12m3.486334788s) [upload=89970]: Upload finished. err: <nil> Duration: 6.137674944s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X.X............................................] duration: 6.009597961s trace: https://jaeger.dev.storj.io/trace/2a65ce5005cafd1d FAIL (2025-11-13 14:50:56 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.00962783s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 2.15642518s trace: https://jaeger.dev.storj.io/trace/d6ec5f1d62f8624 FAIL (2025-11-13 14:50:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:32.932592910 (102h12m31.484063371s) [upload=89979]: Starting upload Nov 13 14:16:32.932874286 (102h12m31.484344767s) [upload=89979]: Done starting upload Nov 13 14:16:32.933042266 (102h12m31.484512717s) [upload=89979]: Uploading segments... Nov 13 14:16:32.933200005 (102h12m31.484670467s) [upload=89979]: Got next segment. Inline: true Nov 13 14:16:32.933208482 (102h12m31.484678943s) [upload=89979]: Waiting for error group managing segments... Nov 13 14:16:32.933309854 (102h12m31.484780316s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:32.933317860 (102h12m31.484788322s) [upload=89979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 13 14:16:32.933321948 (102h12m31.484792409s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:17:01.188559179 (102h12m59.74002964s) [upload=89979]: Done uploading segments... Nov 13 14:17:01.188737939 (102h12m59.7402084s) [upload=89979]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.156454626s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.050657583s trace: https://jaeger.dev.storj.io/trace/d53ad46a3bad038 FAIL (2025-11-13 14:50:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ... Nov 13 14:17:39.718806610 (102h13m38.270277071s) [noise=true , address=5.231.73.3:4030 , piece_id=O4B5VTAWE2WNRG2GZT4OYOEGZLDNYDTHZOT3WETYA3P4XHQR5AOQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.718869189 (102h13m38.27033964s) [noise=true , address=45.91.116.7:4016 , piece_id=BXDTJNCLTSVXQUCC4RO2C2JHZ26XHIJPTHITUO7TXQY6CAY6AC2A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719243782 (102h13m38.270714243s) [noise=true , address=145.79.227.38:4984 , piece_id=LP425WCAJOPGTKLRDOU6V2M3LWRGZ3DZLF4JL4L5QDHW4V37TOZA, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719600372 (102h13m38.271070833s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719757630 (102h13m38.271228102s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719982179 (102h13m38.27145265s) [noise=true , address=5.231.73.2:4018 , piece_id=KP2VQBGZYKUDBJNAIKXV6VRHPK6V3UI6X3SD6HIJOHDTZ6Z6EGLQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720312697 (102h13m38.271783169s) [noise=true , address=77.90.31.17:4776 , piece_id=254SH54AOFXNCTTM6QF3AUN7DYYRE2EKBHDBEUKLHEJMKDBXSVDQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720732096 (102h13m38.272202567s) [noise=true , address=45.91.116.6:4993 , piece_id=B7J3JA5UFDKUW6Q6YACXNQY7EUH7PXYRFUOMMLQKOCWJ6NZ3G3DQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720836485 (102h13m38.272306946s) [noise=true , address=45.91.116.11:4145 , piece_id=7SLQGSEAL7WCGUQ6TRYWNM6F57TZKH6PNDNVXWQGTZ6LKKVXBONA, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.721147126 (102h13m38.272617587s) [noise=true , address=77.90.31.14:4678 , piece_id=D4HEZCFBVRL5CMXNWRCWXFOA3UMT2NXCMIA2L4ACWQ25O3PU4B4A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.721537239 (102h13m38.2730077s) [noise=true , address=5.231.73.4:4022 , piece_id=GABEYHU5QIUMMQYAHKVXLY6QFY4AZRXMVPSDJ3EO6WOIKA7E4U4Q, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.722351200 (102h13m38.273821661s) [noise=true , address=45.91.116.14:4203 , piece_id=SU2IO7NSAAQP4ZEY4QIHXQYOHFFDOSGFQEHFJG44QLIPA37DUA6Q, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.723267113 (102h13m38.274737584s) [noise=true , address=5.231.73.5:4019 , piece_id=H6CZGSTCCYMOZVTSJ2Y5XAKGZA26EQ7E2NONL6DWZGC7Q2LORHBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.723886824 (102h13m38.275357285s) [noise=true , address=64.49.13.26:4948 , piece_id=MQ6RQQQ36YHFSTJYDE44AATC7LQPIFCB7QOS72DMTWRETZVAQWDQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.724315690 (102h13m38.275786151s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.724792528 (102h13m38.276262989s) [noise=true , address=64.49.13.138:4492 , piece_id=CXNCL6OGRP6C4BXBSH7URFSYJELZLM62KVHGCDJQPD7FVTB67Y3A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.725327326 (102h13m38.276797787s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:41.027262394 (102h13m39.578732845s) [upload=89995]: Next returned nil segment Nov 13 14:17:41.027282332 (102h13m39.578752783s) [upload=89995]: Waiting for error group managing segments... Nov 13 14:17:41.357905237 (102h13m39.909375698s) [noise=true , address=64.49.13.26:4948 , piece_id=MQ6RQQQ36YHFSTJYDE44AATC7LQPIFCB7QOS72DMTWRETZVAQWDQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.433145731 (102h13m39.984616192s) [noise=true , address=64.49.13.154:4929 , piece_id=2K55IIOVKBDVMEO3QKXXOZ65X35SQIVSEN6RJAW6OWEGE6ZKYCRA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.460916682 (102h13m40.012387133s) [noise=true , address=5.231.73.2:4018 , piece_id=KP2VQBGZYKUDBJNAIKXV6VRHPK6V3UI6X3SD6HIJOHDTZ6Z6EGLQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.497737346 (102h13m40.049207797s) [noise=true , address=5.231.73.5:4019 , piece_id=H6CZGSTCCYMOZVTSJ2Y5XAKGZA26EQ7E2NONL6DWZGC7Q2LORHBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.511205242 (102h13m40.062675703s) [noise=true , address=5.231.73.3:4030 , piece_id=O4B5VTAWE2WNRG2GZT4OYOEGZLDNYDTHZOT3WETYA3P4XHQR5AOQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.536492093 (102h13m40.087962565s) [noise=true , address=5.231.73.4:4022 , piece_id=GABEYHU5QIUMMQYAHKVXLY6QFY4AZRXMVPSDJ3EO6WOIKA7E4U4Q, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.553999952 (102h13m40.105470433s) [noise=true , address=64.49.13.138:4492 , piece_id=CXNCL6OGRP6C4BXBSH7URFSYJELZLM62KVHGCDJQPD7FVTB67Y3A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.668337736 (102h13m40.219808187s) [noise=true , address=45.91.116.7:4016 , piece_id=BXDTJNCLTSVXQUCC4RO2C2JHZ26XHIJPTHITUO7TXQY6CAY6AC2A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.674376285 (102h13m40.225846746s) [noise=true , address=77.90.31.17:4776 , piece_id=254SH54AOFXNCTTM6QF3AUN7DYYRE2EKBHDBEUKLHEJMKDBXSVDQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.678052054 (102h13m40.229522505s) [noise=true , address=145.79.227.38:4984 , piece_id=LP425WCAJOPGTKLRDOU6V2M3LWRGZ3DZLF4JL4L5QDHW4V37TOZA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.761077612 (102h13m40.312548063s) [noise=true , address=77.90.31.14:4678 , piece_id=D4HEZCFBVRL5CMXNWRCWXFOA3UMT2NXCMIA2L4ACWQ25O3PU4B4A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.771558030 (102h13m40.323028481s) [noise=true , address=45.91.116.6:4993 , piece_id=B7J3JA5UFDKUW6Q6YACXNQY7EUH7PXYRFUOMMLQKOCWJ6NZ3G3DQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.772042953 (102h13m40.323513414s) [noise=true , address=45.91.116.11:4145 , piece_id=7SLQGSEAL7WCGUQ6TRYWNM6F57TZKH6PNDNVXWQGTZ6LKKVXBONA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.775497421 (102h13m40.326967872s) [noise=true , address=45.91.116.14:4203 , piece_id=SU2IO7NSAAQP4ZEY4QIHXQYOHFFDOSGFQEHFJG44QLIPA37DUA6Q, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.775520495 (102h13m40.326990956s) [seg_pos={0 0}, upload=89995]: Segment reached optimal threshold of 14 pieces. Nov 13 14:17:41.775700267 (102h13m40.327170729s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.775707361 (102h13m40.327177812s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.775974550 (102h13m40.327445001s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.775982485 (102h13m40.327452936s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776090991 (102h13m40.327561442s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.776096271 (102h13m40.327566732s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776208986 (102h13m40.327679437s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.776214626 (102h13m40.327685087s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776231869 (102h13m40.32770232s) [seg_pos={0 0}, upload=89995]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 13 14:17:41.776317181 (102h13m40.327787632s) [upload=89995]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:17:41.776322282 (102h13m40.327792743s) [upload=89995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:18:48.053973011 (102h14m46.605443472s) [upload=89995]: Done uploading segments... Nov 13 14:18:48.054067461 (102h14m46.605537912s) [upload=89995]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.050687711s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.702766587s trace: https://jaeger.dev.storj.io/trace/403ba97a5a1cde7d FAIL (2025-11-13 14:50:42 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:00.282283372 (102h11m58.833753833s) [upload=89970]: Starting upload Nov 13 14:16:00.282383773 (102h11m58.833854234s) [upload=89970]: Done starting upload Nov 13 14:16:00.282466601 (102h11m58.833937052s) [upload=89970]: Uploading segments... Nov 13 14:16:00.282681841 (102h11m58.834152302s) [upload=89970]: Got next segment. Inline: false Nov 13 14:16:00.282731475 (102h11m58.834201936s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:00.282737627 (102h11m58.834208088s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 14:16:02.313071205 (102h12m0.864541666s) [seg_pos={0 0}, upload=89970]: Begin upload segment... Nov 13 14:16:02.313149905 (102h12m0.864620376s) [seg_pos={0 0}, upload=89970]: Done begin upload segment. Nov 13 14:16:02.313271697 (102h12m0.864742158s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.313324648 (102h12m0.864795129s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314320904 (102h12m0.865791355s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314744151 (102h12m0.866214642s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315158950 (102h12m0.866629421s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315599659 (102h12m0.867070121s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315928506 (102h12m0.867398967s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.316477040 (102h12m0.867947501s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317049180 (102h12m0.868519651s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317427380 (102h12m0.868897841s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317934746 (102h12m0.869405207s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318376678 (102h12m0.869847139s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318812577 (102h12m0.870283038s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319367303 (102h12m0.870837765s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319957317 (102h12m0.871427768s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.320489491 (102h12m0.871959952s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:03.991369116 (102h12m2.542839567s) [upload=89970]: Next returned nil segment Nov 13 14:16:03.991386198 (102h12m2.542856649s) [upload=89970]: Waiting for error group managing segments... Nov 13 14:16:04.278532114 (102h12m2.830002575s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.279571643 (102h12m2.831042105s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.289684120 (102h12m2.841154581s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.300556964 (102h12m2.852027435s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.301333473 (102h12m2.852803934s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302408349 (102h12m2.853878821s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302631235 (102h12m2.854101716s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.335556962 (102h12m2.887027413s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.346910641 (102h12m2.898381102s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.371810816 (102h12m2.923281277s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377236428 (102h12m2.928706889s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377262828 (102h12m2.928733289s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383693804 (102h12m2.935164265s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383710837 (102h12m2.935181288s) [seg_pos={0 0}, upload=89970]: Segment reached optimal threshold of 13 pieces. Nov 13 14:16:04.383822670 (102h12m2.935293121s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383922450 (102h12m2.935392911s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ): protocol: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection Nov 13 14:16:04.383929954 (102h12m2.935400415s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384043279 (102h12m2.93551374s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection Nov 13 14:16:04.384050112 (102h12m2.935520563s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384066143 (102h12m2.935536604s) [seg_pos={0 0}, upload=89970]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Nov 13 14:16:04.384159561 (102h12m2.935630013s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:16:04.384171734 (102h12m2.935642195s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:16:04.934815134 (102h12m3.486285595s) [upload=89970]: Done uploading segments... Nov 13 14:16:04.934864327 (102h12m3.486334788s) [upload=89970]: Upload finished. err: <nil> Duration: 6.702797046s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.241751536s trace: https://jaeger.dev.storj.io/trace/afd242eeed2927 FAIL (2025-11-13 14:50:27 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.241777556s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 9.457461997s trace: https://jaeger.dev.storj.io/trace/403ba97a5a1cde7d FAIL (2025-11-13 14:50:11 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:00.282283372 (102h11m58.833753833s) [upload=89970]: Starting upload Nov 13 14:16:00.282383773 (102h11m58.833854234s) [upload=89970]: Done starting upload Nov 13 14:16:00.282466601 (102h11m58.833937052s) [upload=89970]: Uploading segments... Nov 13 14:16:00.282681841 (102h11m58.834152302s) [upload=89970]: Got next segment. Inline: false Nov 13 14:16:00.282731475 (102h11m58.834201936s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:00.282737627 (102h11m58.834208088s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 14:16:02.313071205 (102h12m0.864541666s) [seg_pos={0 0}, upload=89970]: Begin upload segment... Nov 13 14:16:02.313149905 (102h12m0.864620376s) [seg_pos={0 0}, upload=89970]: Done begin upload segment. Nov 13 14:16:02.313271697 (102h12m0.864742158s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.313324648 (102h12m0.864795129s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314320904 (102h12m0.865791355s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314744151 (102h12m0.866214642s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315158950 (102h12m0.866629421s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315599659 (102h12m0.867070121s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315928506 (102h12m0.867398967s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.316477040 (102h12m0.867947501s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317049180 (102h12m0.868519651s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317427380 (102h12m0.868897841s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317934746 (102h12m0.869405207s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318376678 (102h12m0.869847139s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318812577 (102h12m0.870283038s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319367303 (102h12m0.870837765s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319957317 (102h12m0.871427768s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.320489491 (102h12m0.871959952s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:03.991369116 (102h12m2.542839567s) [upload=89970]: Next returned nil segment Nov 13 14:16:03.991386198 (102h12m2.542856649s) [upload=89970]: Waiting for error group managing segments... Nov 13 14:16:04.278532114 (102h12m2.830002575s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.279571643 (102h12m2.831042105s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.289684120 (102h12m2.841154581s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.300556964 (102h12m2.852027435s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.301333473 (102h12m2.852803934s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302408349 (102h12m2.853878821s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302631235 (102h12m2.854101716s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.335556962 (102h12m2.887027413s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.346910641 (102h12m2.898381102s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.371810816 (102h12m2.923281277s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377236428 (102h12m2.928706889s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377262828 (102h12m2.928733289s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383693804 (102h12m2.935164265s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383710837 (102h12m2.935181288s) [seg_pos={0 0}, upload=89970]: Segment reached optimal threshold of 13 pieces. Nov 13 14:16:04.383822670 (102h12m2.935293121s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383922450 (102h12m2.935392911s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ): protocol: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection Nov 13 14:16:04.383929954 (102h12m2.935400415s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384043279 (102h12m2.93551374s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection Nov 13 14:16:04.384050112 (102h12m2.935520563s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384066143 (102h12m2.935536604s) [seg_pos={0 0}, upload=89970]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Nov 13 14:16:04.384159561 (102h12m2.935630013s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:16:04.384171734 (102h12m2.935642195s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:16:04.934815134 (102h12m3.486285595s) [upload=89970]: Done uploading segments... Nov 13 14:16:04.934864327 (102h12m3.486334788s) [upload=89970]: Upload finished. err: <nil> Duration: 9.457492996s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.281893267s trace: https://jaeger.dev.storj.io/trace/d6ec5f1d62f8624 FAIL (2025-11-13 14:50:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:32.932592910 (102h12m31.484063371s) [upload=89979]: Starting upload Nov 13 14:16:32.932874286 (102h12m31.484344767s) [upload=89979]: Done starting upload Nov 13 14:16:32.933042266 (102h12m31.484512717s) [upload=89979]: Uploading segments... Nov 13 14:16:32.933200005 (102h12m31.484670467s) [upload=89979]: Got next segment. Inline: true Nov 13 14:16:32.933208482 (102h12m31.484678943s) [upload=89979]: Waiting for error group managing segments... Nov 13 14:16:32.933309854 (102h12m31.484780316s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:32.933317860 (102h12m31.484788322s) [upload=89979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 13 14:16:32.933321948 (102h12m31.484792409s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:17:01.188559179 (102h12m59.74002964s) [upload=89979]: Done uploading segments... Nov 13 14:17:01.188737939 (102h12m59.7402084s) [upload=89979]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.281953201s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.853084402s trace: https://jaeger.dev.storj.io/trace/d53ad46a3bad038 FAIL (2025-11-13 14:50:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ... Nov 13 14:17:39.718806610 (102h13m38.270277071s) [noise=true , address=5.231.73.3:4030 , piece_id=O4B5VTAWE2WNRG2GZT4OYOEGZLDNYDTHZOT3WETYA3P4XHQR5AOQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.718869189 (102h13m38.27033964s) [noise=true , address=45.91.116.7:4016 , piece_id=BXDTJNCLTSVXQUCC4RO2C2JHZ26XHIJPTHITUO7TXQY6CAY6AC2A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719243782 (102h13m38.270714243s) [noise=true , address=145.79.227.38:4984 , piece_id=LP425WCAJOPGTKLRDOU6V2M3LWRGZ3DZLF4JL4L5QDHW4V37TOZA, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719600372 (102h13m38.271070833s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719757630 (102h13m38.271228102s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719982179 (102h13m38.27145265s) [noise=true , address=5.231.73.2:4018 , piece_id=KP2VQBGZYKUDBJNAIKXV6VRHPK6V3UI6X3SD6HIJOHDTZ6Z6EGLQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720312697 (102h13m38.271783169s) [noise=true , address=77.90.31.17:4776 , piece_id=254SH54AOFXNCTTM6QF3AUN7DYYRE2EKBHDBEUKLHEJMKDBXSVDQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720732096 (102h13m38.272202567s) [noise=true , address=45.91.116.6:4993 , piece_id=B7J3JA5UFDKUW6Q6YACXNQY7EUH7PXYRFUOMMLQKOCWJ6NZ3G3DQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720836485 (102h13m38.272306946s) [noise=true , address=45.91.116.11:4145 , piece_id=7SLQGSEAL7WCGUQ6TRYWNM6F57TZKH6PNDNVXWQGTZ6LKKVXBONA, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.721147126 (102h13m38.272617587s) [noise=true , address=77.90.31.14:4678 , piece_id=D4HEZCFBVRL5CMXNWRCWXFOA3UMT2NXCMIA2L4ACWQ25O3PU4B4A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.721537239 (102h13m38.2730077s) [noise=true , address=5.231.73.4:4022 , piece_id=GABEYHU5QIUMMQYAHKVXLY6QFY4AZRXMVPSDJ3EO6WOIKA7E4U4Q, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.722351200 (102h13m38.273821661s) [noise=true , address=45.91.116.14:4203 , piece_id=SU2IO7NSAAQP4ZEY4QIHXQYOHFFDOSGFQEHFJG44QLIPA37DUA6Q, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.723267113 (102h13m38.274737584s) [noise=true , address=5.231.73.5:4019 , piece_id=H6CZGSTCCYMOZVTSJ2Y5XAKGZA26EQ7E2NONL6DWZGC7Q2LORHBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.723886824 (102h13m38.275357285s) [noise=true , address=64.49.13.26:4948 , piece_id=MQ6RQQQ36YHFSTJYDE44AATC7LQPIFCB7QOS72DMTWRETZVAQWDQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.724315690 (102h13m38.275786151s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.724792528 (102h13m38.276262989s) [noise=true , address=64.49.13.138:4492 , piece_id=CXNCL6OGRP6C4BXBSH7URFSYJELZLM62KVHGCDJQPD7FVTB67Y3A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.725327326 (102h13m38.276797787s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:41.027262394 (102h13m39.578732845s) [upload=89995]: Next returned nil segment Nov 13 14:17:41.027282332 (102h13m39.578752783s) [upload=89995]: Waiting for error group managing segments... Nov 13 14:17:41.357905237 (102h13m39.909375698s) [noise=true , address=64.49.13.26:4948 , piece_id=MQ6RQQQ36YHFSTJYDE44AATC7LQPIFCB7QOS72DMTWRETZVAQWDQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.433145731 (102h13m39.984616192s) [noise=true , address=64.49.13.154:4929 , piece_id=2K55IIOVKBDVMEO3QKXXOZ65X35SQIVSEN6RJAW6OWEGE6ZKYCRA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.460916682 (102h13m40.012387133s) [noise=true , address=5.231.73.2:4018 , piece_id=KP2VQBGZYKUDBJNAIKXV6VRHPK6V3UI6X3SD6HIJOHDTZ6Z6EGLQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.497737346 (102h13m40.049207797s) [noise=true , address=5.231.73.5:4019 , piece_id=H6CZGSTCCYMOZVTSJ2Y5XAKGZA26EQ7E2NONL6DWZGC7Q2LORHBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.511205242 (102h13m40.062675703s) [noise=true , address=5.231.73.3:4030 , piece_id=O4B5VTAWE2WNRG2GZT4OYOEGZLDNYDTHZOT3WETYA3P4XHQR5AOQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.536492093 (102h13m40.087962565s) [noise=true , address=5.231.73.4:4022 , piece_id=GABEYHU5QIUMMQYAHKVXLY6QFY4AZRXMVPSDJ3EO6WOIKA7E4U4Q, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.553999952 (102h13m40.105470433s) [noise=true , address=64.49.13.138:4492 , piece_id=CXNCL6OGRP6C4BXBSH7URFSYJELZLM62KVHGCDJQPD7FVTB67Y3A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.668337736 (102h13m40.219808187s) [noise=true , address=45.91.116.7:4016 , piece_id=BXDTJNCLTSVXQUCC4RO2C2JHZ26XHIJPTHITUO7TXQY6CAY6AC2A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.674376285 (102h13m40.225846746s) [noise=true , address=77.90.31.17:4776 , piece_id=254SH54AOFXNCTTM6QF3AUN7DYYRE2EKBHDBEUKLHEJMKDBXSVDQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.678052054 (102h13m40.229522505s) [noise=true , address=145.79.227.38:4984 , piece_id=LP425WCAJOPGTKLRDOU6V2M3LWRGZ3DZLF4JL4L5QDHW4V37TOZA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.761077612 (102h13m40.312548063s) [noise=true , address=77.90.31.14:4678 , piece_id=D4HEZCFBVRL5CMXNWRCWXFOA3UMT2NXCMIA2L4ACWQ25O3PU4B4A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.771558030 (102h13m40.323028481s) [noise=true , address=45.91.116.6:4993 , piece_id=B7J3JA5UFDKUW6Q6YACXNQY7EUH7PXYRFUOMMLQKOCWJ6NZ3G3DQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.772042953 (102h13m40.323513414s) [noise=true , address=45.91.116.11:4145 , piece_id=7SLQGSEAL7WCGUQ6TRYWNM6F57TZKH6PNDNVXWQGTZ6LKKVXBONA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.775497421 (102h13m40.326967872s) [noise=true , address=45.91.116.14:4203 , piece_id=SU2IO7NSAAQP4ZEY4QIHXQYOHFFDOSGFQEHFJG44QLIPA37DUA6Q, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.775520495 (102h13m40.326990956s) [seg_pos={0 0}, upload=89995]: Segment reached optimal threshold of 14 pieces. Nov 13 14:17:41.775700267 (102h13m40.327170729s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.775707361 (102h13m40.327177812s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.775974550 (102h13m40.327445001s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.775982485 (102h13m40.327452936s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776090991 (102h13m40.327561442s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.776096271 (102h13m40.327566732s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776208986 (102h13m40.327679437s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.776214626 (102h13m40.327685087s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776231869 (102h13m40.32770232s) [seg_pos={0 0}, upload=89995]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 13 14:17:41.776317181 (102h13m40.327787632s) [upload=89995]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:17:41.776322282 (102h13m40.327792743s) [upload=89995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:18:48.053973011 (102h14m46.605443472s) [upload=89995]: Done uploading segments... Nov 13 14:18:48.054067461 (102h14m46.605537912s) [upload=89995]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.853119339s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.130467146s trace: https://jaeger.dev.storj.io/trace/afd242eeed2927 FAIL (2025-11-13 14:49:51 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.130482256s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X.X...........................................] duration: 6.302842337s trace: https://jaeger.dev.storj.io/trace/2a65ce5005cafd1d FAIL (2025-11-13 14:49:49 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.302868748s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.122081079s trace: https://jaeger.dev.storj.io/trace/d6ec5f1d62f8624 FAIL (2025-11-13 14:49:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:32.932592910 (102h12m31.484063371s) [upload=89979]: Starting upload Nov 13 14:16:32.932874286 (102h12m31.484344767s) [upload=89979]: Done starting upload Nov 13 14:16:32.933042266 (102h12m31.484512717s) [upload=89979]: Uploading segments... Nov 13 14:16:32.933200005 (102h12m31.484670467s) [upload=89979]: Got next segment. Inline: true Nov 13 14:16:32.933208482 (102h12m31.484678943s) [upload=89979]: Waiting for error group managing segments... Nov 13 14:16:32.933309854 (102h12m31.484780316s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:32.933317860 (102h12m31.484788322s) [upload=89979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 13 14:16:32.933321948 (102h12m31.484792409s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:17:01.188559179 (102h12m59.74002964s) [upload=89979]: Done uploading segments... Nov 13 14:17:01.188737939 (102h12m59.7402084s) [upload=89979]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.122096688s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.482292982s trace: https://jaeger.dev.storj.io/trace/d53ad46a3bad038 FAIL (2025-11-13 14:49:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ... Nov 13 14:17:39.718806610 (102h13m38.270277071s) [noise=true , address=5.231.73.3:4030 , piece_id=O4B5VTAWE2WNRG2GZT4OYOEGZLDNYDTHZOT3WETYA3P4XHQR5AOQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.718869189 (102h13m38.27033964s) [noise=true , address=45.91.116.7:4016 , piece_id=BXDTJNCLTSVXQUCC4RO2C2JHZ26XHIJPTHITUO7TXQY6CAY6AC2A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719243782 (102h13m38.270714243s) [noise=true , address=145.79.227.38:4984 , piece_id=LP425WCAJOPGTKLRDOU6V2M3LWRGZ3DZLF4JL4L5QDHW4V37TOZA, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719600372 (102h13m38.271070833s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719757630 (102h13m38.271228102s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719982179 (102h13m38.27145265s) [noise=true , address=5.231.73.2:4018 , piece_id=KP2VQBGZYKUDBJNAIKXV6VRHPK6V3UI6X3SD6HIJOHDTZ6Z6EGLQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720312697 (102h13m38.271783169s) [noise=true , address=77.90.31.17:4776 , piece_id=254SH54AOFXNCTTM6QF3AUN7DYYRE2EKBHDBEUKLHEJMKDBXSVDQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720732096 (102h13m38.272202567s) [noise=true , address=45.91.116.6:4993 , piece_id=B7J3JA5UFDKUW6Q6YACXNQY7EUH7PXYRFUOMMLQKOCWJ6NZ3G3DQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720836485 (102h13m38.272306946s) [noise=true , address=45.91.116.11:4145 , piece_id=7SLQGSEAL7WCGUQ6TRYWNM6F57TZKH6PNDNVXWQGTZ6LKKVXBONA, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.721147126 (102h13m38.272617587s) [noise=true , address=77.90.31.14:4678 , piece_id=D4HEZCFBVRL5CMXNWRCWXFOA3UMT2NXCMIA2L4ACWQ25O3PU4B4A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.721537239 (102h13m38.2730077s) [noise=true , address=5.231.73.4:4022 , piece_id=GABEYHU5QIUMMQYAHKVXLY6QFY4AZRXMVPSDJ3EO6WOIKA7E4U4Q, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.722351200 (102h13m38.273821661s) [noise=true , address=45.91.116.14:4203 , piece_id=SU2IO7NSAAQP4ZEY4QIHXQYOHFFDOSGFQEHFJG44QLIPA37DUA6Q, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.723267113 (102h13m38.274737584s) [noise=true , address=5.231.73.5:4019 , piece_id=H6CZGSTCCYMOZVTSJ2Y5XAKGZA26EQ7E2NONL6DWZGC7Q2LORHBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.723886824 (102h13m38.275357285s) [noise=true , address=64.49.13.26:4948 , piece_id=MQ6RQQQ36YHFSTJYDE44AATC7LQPIFCB7QOS72DMTWRETZVAQWDQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.724315690 (102h13m38.275786151s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.724792528 (102h13m38.276262989s) [noise=true , address=64.49.13.138:4492 , piece_id=CXNCL6OGRP6C4BXBSH7URFSYJELZLM62KVHGCDJQPD7FVTB67Y3A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.725327326 (102h13m38.276797787s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:41.027262394 (102h13m39.578732845s) [upload=89995]: Next returned nil segment Nov 13 14:17:41.027282332 (102h13m39.578752783s) [upload=89995]: Waiting for error group managing segments... Nov 13 14:17:41.357905237 (102h13m39.909375698s) [noise=true , address=64.49.13.26:4948 , piece_id=MQ6RQQQ36YHFSTJYDE44AATC7LQPIFCB7QOS72DMTWRETZVAQWDQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.433145731 (102h13m39.984616192s) [noise=true , address=64.49.13.154:4929 , piece_id=2K55IIOVKBDVMEO3QKXXOZ65X35SQIVSEN6RJAW6OWEGE6ZKYCRA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.460916682 (102h13m40.012387133s) [noise=true , address=5.231.73.2:4018 , piece_id=KP2VQBGZYKUDBJNAIKXV6VRHPK6V3UI6X3SD6HIJOHDTZ6Z6EGLQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.497737346 (102h13m40.049207797s) [noise=true , address=5.231.73.5:4019 , piece_id=H6CZGSTCCYMOZVTSJ2Y5XAKGZA26EQ7E2NONL6DWZGC7Q2LORHBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.511205242 (102h13m40.062675703s) [noise=true , address=5.231.73.3:4030 , piece_id=O4B5VTAWE2WNRG2GZT4OYOEGZLDNYDTHZOT3WETYA3P4XHQR5AOQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.536492093 (102h13m40.087962565s) [noise=true , address=5.231.73.4:4022 , piece_id=GABEYHU5QIUMMQYAHKVXLY6QFY4AZRXMVPSDJ3EO6WOIKA7E4U4Q, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.553999952 (102h13m40.105470433s) [noise=true , address=64.49.13.138:4492 , piece_id=CXNCL6OGRP6C4BXBSH7URFSYJELZLM62KVHGCDJQPD7FVTB67Y3A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.668337736 (102h13m40.219808187s) [noise=true , address=45.91.116.7:4016 , piece_id=BXDTJNCLTSVXQUCC4RO2C2JHZ26XHIJPTHITUO7TXQY6CAY6AC2A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.674376285 (102h13m40.225846746s) [noise=true , address=77.90.31.17:4776 , piece_id=254SH54AOFXNCTTM6QF3AUN7DYYRE2EKBHDBEUKLHEJMKDBXSVDQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.678052054 (102h13m40.229522505s) [noise=true , address=145.79.227.38:4984 , piece_id=LP425WCAJOPGTKLRDOU6V2M3LWRGZ3DZLF4JL4L5QDHW4V37TOZA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.761077612 (102h13m40.312548063s) [noise=true , address=77.90.31.14:4678 , piece_id=D4HEZCFBVRL5CMXNWRCWXFOA3UMT2NXCMIA2L4ACWQ25O3PU4B4A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.771558030 (102h13m40.323028481s) [noise=true , address=45.91.116.6:4993 , piece_id=B7J3JA5UFDKUW6Q6YACXNQY7EUH7PXYRFUOMMLQKOCWJ6NZ3G3DQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.772042953 (102h13m40.323513414s) [noise=true , address=45.91.116.11:4145 , piece_id=7SLQGSEAL7WCGUQ6TRYWNM6F57TZKH6PNDNVXWQGTZ6LKKVXBONA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.775497421 (102h13m40.326967872s) [noise=true , address=45.91.116.14:4203 , piece_id=SU2IO7NSAAQP4ZEY4QIHXQYOHFFDOSGFQEHFJG44QLIPA37DUA6Q, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.775520495 (102h13m40.326990956s) [seg_pos={0 0}, upload=89995]: Segment reached optimal threshold of 14 pieces. Nov 13 14:17:41.775700267 (102h13m40.327170729s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.775707361 (102h13m40.327177812s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.775974550 (102h13m40.327445001s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.775982485 (102h13m40.327452936s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776090991 (102h13m40.327561442s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.776096271 (102h13m40.327566732s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776208986 (102h13m40.327679437s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.776214626 (102h13m40.327685087s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776231869 (102h13m40.32770232s) [seg_pos={0 0}, upload=89995]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 13 14:17:41.776317181 (102h13m40.327787632s) [upload=89995]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:17:41.776322282 (102h13m40.327792743s) [upload=89995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:18:48.053973011 (102h14m46.605443472s) [upload=89995]: Done uploading segments... Nov 13 14:18:48.054067461 (102h14m46.605537912s) [upload=89995]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.482413913s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.174084095s trace: https://jaeger.dev.storj.io/trace/d6ec5f1d62f8624 FAIL (2025-11-13 14:49:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:32.932592910 (102h12m31.484063371s) [upload=89979]: Starting upload Nov 13 14:16:32.932874286 (102h12m31.484344767s) [upload=89979]: Done starting upload Nov 13 14:16:32.933042266 (102h12m31.484512717s) [upload=89979]: Uploading segments... Nov 13 14:16:32.933200005 (102h12m31.484670467s) [upload=89979]: Got next segment. Inline: true Nov 13 14:16:32.933208482 (102h12m31.484678943s) [upload=89979]: Waiting for error group managing segments... Nov 13 14:16:32.933309854 (102h12m31.484780316s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:32.933317860 (102h12m31.484788322s) [upload=89979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 13 14:16:32.933321948 (102h12m31.484792409s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:17:01.188559179 (102h12m59.74002964s) [upload=89979]: Done uploading segments... Nov 13 14:17:01.188737939 (102h12m59.7402084s) [upload=89979]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.174115164s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.265848422s trace: https://jaeger.dev.storj.io/trace/afd242eeed2927 FAIL (2025-11-13 14:49:03 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.265949696s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.547471525s trace: https://jaeger.dev.storj.io/trace/403ba97a5a1cde7d FAIL (2025-11-13 14:49:01 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:00.282283372 (102h11m58.833753833s) [upload=89970]: Starting upload Nov 13 14:16:00.282383773 (102h11m58.833854234s) [upload=89970]: Done starting upload Nov 13 14:16:00.282466601 (102h11m58.833937052s) [upload=89970]: Uploading segments... Nov 13 14:16:00.282681841 (102h11m58.834152302s) [upload=89970]: Got next segment. Inline: false Nov 13 14:16:00.282731475 (102h11m58.834201936s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:00.282737627 (102h11m58.834208088s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 14:16:02.313071205 (102h12m0.864541666s) [seg_pos={0 0}, upload=89970]: Begin upload segment... Nov 13 14:16:02.313149905 (102h12m0.864620376s) [seg_pos={0 0}, upload=89970]: Done begin upload segment. Nov 13 14:16:02.313271697 (102h12m0.864742158s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.313324648 (102h12m0.864795129s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314320904 (102h12m0.865791355s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314744151 (102h12m0.866214642s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315158950 (102h12m0.866629421s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315599659 (102h12m0.867070121s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315928506 (102h12m0.867398967s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.316477040 (102h12m0.867947501s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317049180 (102h12m0.868519651s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317427380 (102h12m0.868897841s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317934746 (102h12m0.869405207s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318376678 (102h12m0.869847139s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318812577 (102h12m0.870283038s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319367303 (102h12m0.870837765s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319957317 (102h12m0.871427768s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.320489491 (102h12m0.871959952s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:03.991369116 (102h12m2.542839567s) [upload=89970]: Next returned nil segment Nov 13 14:16:03.991386198 (102h12m2.542856649s) [upload=89970]: Waiting for error group managing segments... Nov 13 14:16:04.278532114 (102h12m2.830002575s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.279571643 (102h12m2.831042105s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.289684120 (102h12m2.841154581s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.300556964 (102h12m2.852027435s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.301333473 (102h12m2.852803934s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302408349 (102h12m2.853878821s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302631235 (102h12m2.854101716s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.335556962 (102h12m2.887027413s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.346910641 (102h12m2.898381102s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.371810816 (102h12m2.923281277s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377236428 (102h12m2.928706889s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377262828 (102h12m2.928733289s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383693804 (102h12m2.935164265s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383710837 (102h12m2.935181288s) [seg_pos={0 0}, upload=89970]: Segment reached optimal threshold of 13 pieces. Nov 13 14:16:04.383822670 (102h12m2.935293121s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383922450 (102h12m2.935392911s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ): protocol: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection Nov 13 14:16:04.383929954 (102h12m2.935400415s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384043279 (102h12m2.93551374s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection Nov 13 14:16:04.384050112 (102h12m2.935520563s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384066143 (102h12m2.935536604s) [seg_pos={0 0}, upload=89970]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Nov 13 14:16:04.384159561 (102h12m2.935630013s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:16:04.384171734 (102h12m2.935642195s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:16:04.934815134 (102h12m3.486285595s) [upload=89970]: Done uploading segments... Nov 13 14:16:04.934864327 (102h12m3.486334788s) [upload=89970]: Upload finished. err: <nil> Duration: 6.547502424s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.938036069s trace: https://jaeger.dev.storj.io/trace/d53ad46a3bad038 FAIL (2025-11-13 14:48:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ... Nov 13 14:17:39.718806610 (102h13m38.270277071s) [noise=true , address=5.231.73.3:4030 , piece_id=O4B5VTAWE2WNRG2GZT4OYOEGZLDNYDTHZOT3WETYA3P4XHQR5AOQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.718869189 (102h13m38.27033964s) [noise=true , address=45.91.116.7:4016 , piece_id=BXDTJNCLTSVXQUCC4RO2C2JHZ26XHIJPTHITUO7TXQY6CAY6AC2A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719243782 (102h13m38.270714243s) [noise=true , address=145.79.227.38:4984 , piece_id=LP425WCAJOPGTKLRDOU6V2M3LWRGZ3DZLF4JL4L5QDHW4V37TOZA, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719600372 (102h13m38.271070833s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719757630 (102h13m38.271228102s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719982179 (102h13m38.27145265s) [noise=true , address=5.231.73.2:4018 , piece_id=KP2VQBGZYKUDBJNAIKXV6VRHPK6V3UI6X3SD6HIJOHDTZ6Z6EGLQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720312697 (102h13m38.271783169s) [noise=true , address=77.90.31.17:4776 , piece_id=254SH54AOFXNCTTM6QF3AUN7DYYRE2EKBHDBEUKLHEJMKDBXSVDQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720732096 (102h13m38.272202567s) [noise=true , address=45.91.116.6:4993 , piece_id=B7J3JA5UFDKUW6Q6YACXNQY7EUH7PXYRFUOMMLQKOCWJ6NZ3G3DQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720836485 (102h13m38.272306946s) [noise=true , address=45.91.116.11:4145 , piece_id=7SLQGSEAL7WCGUQ6TRYWNM6F57TZKH6PNDNVXWQGTZ6LKKVXBONA, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.721147126 (102h13m38.272617587s) [noise=true , address=77.90.31.14:4678 , piece_id=D4HEZCFBVRL5CMXNWRCWXFOA3UMT2NXCMIA2L4ACWQ25O3PU4B4A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.721537239 (102h13m38.2730077s) [noise=true , address=5.231.73.4:4022 , piece_id=GABEYHU5QIUMMQYAHKVXLY6QFY4AZRXMVPSDJ3EO6WOIKA7E4U4Q, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.722351200 (102h13m38.273821661s) [noise=true , address=45.91.116.14:4203 , piece_id=SU2IO7NSAAQP4ZEY4QIHXQYOHFFDOSGFQEHFJG44QLIPA37DUA6Q, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.723267113 (102h13m38.274737584s) [noise=true , address=5.231.73.5:4019 , piece_id=H6CZGSTCCYMOZVTSJ2Y5XAKGZA26EQ7E2NONL6DWZGC7Q2LORHBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.723886824 (102h13m38.275357285s) [noise=true , address=64.49.13.26:4948 , piece_id=MQ6RQQQ36YHFSTJYDE44AATC7LQPIFCB7QOS72DMTWRETZVAQWDQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.724315690 (102h13m38.275786151s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.724792528 (102h13m38.276262989s) [noise=true , address=64.49.13.138:4492 , piece_id=CXNCL6OGRP6C4BXBSH7URFSYJELZLM62KVHGCDJQPD7FVTB67Y3A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.725327326 (102h13m38.276797787s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:41.027262394 (102h13m39.578732845s) [upload=89995]: Next returned nil segment Nov 13 14:17:41.027282332 (102h13m39.578752783s) [upload=89995]: Waiting for error group managing segments... Nov 13 14:17:41.357905237 (102h13m39.909375698s) [noise=true , address=64.49.13.26:4948 , piece_id=MQ6RQQQ36YHFSTJYDE44AATC7LQPIFCB7QOS72DMTWRETZVAQWDQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.433145731 (102h13m39.984616192s) [noise=true , address=64.49.13.154:4929 , piece_id=2K55IIOVKBDVMEO3QKXXOZ65X35SQIVSEN6RJAW6OWEGE6ZKYCRA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.460916682 (102h13m40.012387133s) [noise=true , address=5.231.73.2:4018 , piece_id=KP2VQBGZYKUDBJNAIKXV6VRHPK6V3UI6X3SD6HIJOHDTZ6Z6EGLQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.497737346 (102h13m40.049207797s) [noise=true , address=5.231.73.5:4019 , piece_id=H6CZGSTCCYMOZVTSJ2Y5XAKGZA26EQ7E2NONL6DWZGC7Q2LORHBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.511205242 (102h13m40.062675703s) [noise=true , address=5.231.73.3:4030 , piece_id=O4B5VTAWE2WNRG2GZT4OYOEGZLDNYDTHZOT3WETYA3P4XHQR5AOQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.536492093 (102h13m40.087962565s) [noise=true , address=5.231.73.4:4022 , piece_id=GABEYHU5QIUMMQYAHKVXLY6QFY4AZRXMVPSDJ3EO6WOIKA7E4U4Q, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.553999952 (102h13m40.105470433s) [noise=true , address=64.49.13.138:4492 , piece_id=CXNCL6OGRP6C4BXBSH7URFSYJELZLM62KVHGCDJQPD7FVTB67Y3A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.668337736 (102h13m40.219808187s) [noise=true , address=45.91.116.7:4016 , piece_id=BXDTJNCLTSVXQUCC4RO2C2JHZ26XHIJPTHITUO7TXQY6CAY6AC2A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.674376285 (102h13m40.225846746s) [noise=true , address=77.90.31.17:4776 , piece_id=254SH54AOFXNCTTM6QF3AUN7DYYRE2EKBHDBEUKLHEJMKDBXSVDQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.678052054 (102h13m40.229522505s) [noise=true , address=145.79.227.38:4984 , piece_id=LP425WCAJOPGTKLRDOU6V2M3LWRGZ3DZLF4JL4L5QDHW4V37TOZA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.761077612 (102h13m40.312548063s) [noise=true , address=77.90.31.14:4678 , piece_id=D4HEZCFBVRL5CMXNWRCWXFOA3UMT2NXCMIA2L4ACWQ25O3PU4B4A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.771558030 (102h13m40.323028481s) [noise=true , address=45.91.116.6:4993 , piece_id=B7J3JA5UFDKUW6Q6YACXNQY7EUH7PXYRFUOMMLQKOCWJ6NZ3G3DQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.772042953 (102h13m40.323513414s) [noise=true , address=45.91.116.11:4145 , piece_id=7SLQGSEAL7WCGUQ6TRYWNM6F57TZKH6PNDNVXWQGTZ6LKKVXBONA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.775497421 (102h13m40.326967872s) [noise=true , address=45.91.116.14:4203 , piece_id=SU2IO7NSAAQP4ZEY4QIHXQYOHFFDOSGFQEHFJG44QLIPA37DUA6Q, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.775520495 (102h13m40.326990956s) [seg_pos={0 0}, upload=89995]: Segment reached optimal threshold of 14 pieces. Nov 13 14:17:41.775700267 (102h13m40.327170729s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.775707361 (102h13m40.327177812s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.775974550 (102h13m40.327445001s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.775982485 (102h13m40.327452936s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776090991 (102h13m40.327561442s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.776096271 (102h13m40.327566732s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776208986 (102h13m40.327679437s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.776214626 (102h13m40.327685087s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776231869 (102h13m40.32770232s) [seg_pos={0 0}, upload=89995]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 13 14:17:41.776317181 (102h13m40.327787632s) [upload=89995]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:17:41.776322282 (102h13m40.327792743s) [upload=89995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:18:48.053973011 (102h14m46.605443472s) [upload=89995]: Done uploading segments... Nov 13 14:18:48.054067461 (102h14m46.605537912s) [upload=89995]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.93817177s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X.X..........................................] duration: 6.005904128s trace: https://jaeger.dev.storj.io/trace/2a65ce5005cafd1d FAIL (2025-11-13 14:48:49 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.005935436s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.165889701s trace: https://jaeger.dev.storj.io/trace/afd242eeed2927 FAIL (2025-11-13 14:48:29 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.165921722s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.202144085s trace: https://jaeger.dev.storj.io/trace/d6ec5f1d62f8624 FAIL (2025-11-13 14:48:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:32.932592910 (102h12m31.484063371s) [upload=89979]: Starting upload Nov 13 14:16:32.932874286 (102h12m31.484344767s) [upload=89979]: Done starting upload Nov 13 14:16:32.933042266 (102h12m31.484512717s) [upload=89979]: Uploading segments... Nov 13 14:16:32.933200005 (102h12m31.484670467s) [upload=89979]: Got next segment. Inline: true Nov 13 14:16:32.933208482 (102h12m31.484678943s) [upload=89979]: Waiting for error group managing segments... Nov 13 14:16:32.933309854 (102h12m31.484780316s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:32.933317860 (102h12m31.484788322s) [upload=89979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 13 14:16:32.933321948 (102h12m31.484792409s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:17:01.188559179 (102h12m59.74002964s) [upload=89979]: Done uploading segments... Nov 13 14:17:01.188737939 (102h12m59.7402084s) [upload=89979]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.202176659s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.165920429s trace: https://jaeger.dev.storj.io/trace/afd242eeed2927 FAIL (2025-11-13 14:48:03 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.165950917s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 14.810816871s trace: https://jaeger.dev.storj.io/trace/403ba97a5a1cde7d FAIL (2025-11-13 14:48:02 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:00.282283372 (102h11m58.833753833s) [upload=89970]: Starting upload Nov 13 14:16:00.282383773 (102h11m58.833854234s) [upload=89970]: Done starting upload Nov 13 14:16:00.282466601 (102h11m58.833937052s) [upload=89970]: Uploading segments... Nov 13 14:16:00.282681841 (102h11m58.834152302s) [upload=89970]: Got next segment. Inline: false Nov 13 14:16:00.282731475 (102h11m58.834201936s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:00.282737627 (102h11m58.834208088s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 14:16:02.313071205 (102h12m0.864541666s) [seg_pos={0 0}, upload=89970]: Begin upload segment... Nov 13 14:16:02.313149905 (102h12m0.864620376s) [seg_pos={0 0}, upload=89970]: Done begin upload segment. Nov 13 14:16:02.313271697 (102h12m0.864742158s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.313324648 (102h12m0.864795129s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314320904 (102h12m0.865791355s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314744151 (102h12m0.866214642s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315158950 (102h12m0.866629421s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315599659 (102h12m0.867070121s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315928506 (102h12m0.867398967s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.316477040 (102h12m0.867947501s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317049180 (102h12m0.868519651s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317427380 (102h12m0.868897841s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317934746 (102h12m0.869405207s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318376678 (102h12m0.869847139s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318812577 (102h12m0.870283038s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319367303 (102h12m0.870837765s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319957317 (102h12m0.871427768s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.320489491 (102h12m0.871959952s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:03.991369116 (102h12m2.542839567s) [upload=89970]: Next returned nil segment Nov 13 14:16:03.991386198 (102h12m2.542856649s) [upload=89970]: Waiting for error group managing segments... Nov 13 14:16:04.278532114 (102h12m2.830002575s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.279571643 (102h12m2.831042105s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.289684120 (102h12m2.841154581s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.300556964 (102h12m2.852027435s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.301333473 (102h12m2.852803934s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302408349 (102h12m2.853878821s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302631235 (102h12m2.854101716s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.335556962 (102h12m2.887027413s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.346910641 (102h12m2.898381102s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.371810816 (102h12m2.923281277s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377236428 (102h12m2.928706889s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377262828 (102h12m2.928733289s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383693804 (102h12m2.935164265s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383710837 (102h12m2.935181288s) [seg_pos={0 0}, upload=89970]: Segment reached optimal threshold of 13 pieces. Nov 13 14:16:04.383822670 (102h12m2.935293121s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383922450 (102h12m2.935392911s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ): protocol: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection Nov 13 14:16:04.383929954 (102h12m2.935400415s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384043279 (102h12m2.93551374s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection Nov 13 14:16:04.384050112 (102h12m2.935520563s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384066143 (102h12m2.935536604s) [seg_pos={0 0}, upload=89970]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Nov 13 14:16:04.384159561 (102h12m2.935630013s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:16:04.384171734 (102h12m2.935642195s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:16:04.934815134 (102h12m3.486285595s) [upload=89970]: Done uploading segments... Nov 13 14:16:04.934864327 (102h12m3.486334788s) [upload=89970]: Upload finished. err: <nil> Duration: 14.810897926s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.293779918s trace: https://jaeger.dev.storj.io/trace/d53ad46a3bad038 FAIL (2025-11-13 14:47:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ... Nov 13 14:17:39.718806610 (102h13m38.270277071s) [noise=true , address=5.231.73.3:4030 , piece_id=O4B5VTAWE2WNRG2GZT4OYOEGZLDNYDTHZOT3WETYA3P4XHQR5AOQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.718869189 (102h13m38.27033964s) [noise=true , address=45.91.116.7:4016 , piece_id=BXDTJNCLTSVXQUCC4RO2C2JHZ26XHIJPTHITUO7TXQY6CAY6AC2A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719243782 (102h13m38.270714243s) [noise=true , address=145.79.227.38:4984 , piece_id=LP425WCAJOPGTKLRDOU6V2M3LWRGZ3DZLF4JL4L5QDHW4V37TOZA, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719600372 (102h13m38.271070833s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719757630 (102h13m38.271228102s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719982179 (102h13m38.27145265s) [noise=true , address=5.231.73.2:4018 , piece_id=KP2VQBGZYKUDBJNAIKXV6VRHPK6V3UI6X3SD6HIJOHDTZ6Z6EGLQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720312697 (102h13m38.271783169s) [noise=true , address=77.90.31.17:4776 , piece_id=254SH54AOFXNCTTM6QF3AUN7DYYRE2EKBHDBEUKLHEJMKDBXSVDQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720732096 (102h13m38.272202567s) [noise=true , address=45.91.116.6:4993 , piece_id=B7J3JA5UFDKUW6Q6YACXNQY7EUH7PXYRFUOMMLQKOCWJ6NZ3G3DQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720836485 (102h13m38.272306946s) [noise=true , address=45.91.116.11:4145 , piece_id=7SLQGSEAL7WCGUQ6TRYWNM6F57TZKH6PNDNVXWQGTZ6LKKVXBONA, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.721147126 (102h13m38.272617587s) [noise=true , address=77.90.31.14:4678 , piece_id=D4HEZCFBVRL5CMXNWRCWXFOA3UMT2NXCMIA2L4ACWQ25O3PU4B4A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.721537239 (102h13m38.2730077s) [noise=true , address=5.231.73.4:4022 , piece_id=GABEYHU5QIUMMQYAHKVXLY6QFY4AZRXMVPSDJ3EO6WOIKA7E4U4Q, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.722351200 (102h13m38.273821661s) [noise=true , address=45.91.116.14:4203 , piece_id=SU2IO7NSAAQP4ZEY4QIHXQYOHFFDOSGFQEHFJG44QLIPA37DUA6Q, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.723267113 (102h13m38.274737584s) [noise=true , address=5.231.73.5:4019 , piece_id=H6CZGSTCCYMOZVTSJ2Y5XAKGZA26EQ7E2NONL6DWZGC7Q2LORHBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.723886824 (102h13m38.275357285s) [noise=true , address=64.49.13.26:4948 , piece_id=MQ6RQQQ36YHFSTJYDE44AATC7LQPIFCB7QOS72DMTWRETZVAQWDQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.724315690 (102h13m38.275786151s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.724792528 (102h13m38.276262989s) [noise=true , address=64.49.13.138:4492 , piece_id=CXNCL6OGRP6C4BXBSH7URFSYJELZLM62KVHGCDJQPD7FVTB67Y3A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.725327326 (102h13m38.276797787s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:41.027262394 (102h13m39.578732845s) [upload=89995]: Next returned nil segment Nov 13 14:17:41.027282332 (102h13m39.578752783s) [upload=89995]: Waiting for error group managing segments... Nov 13 14:17:41.357905237 (102h13m39.909375698s) [noise=true , address=64.49.13.26:4948 , piece_id=MQ6RQQQ36YHFSTJYDE44AATC7LQPIFCB7QOS72DMTWRETZVAQWDQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.433145731 (102h13m39.984616192s) [noise=true , address=64.49.13.154:4929 , piece_id=2K55IIOVKBDVMEO3QKXXOZ65X35SQIVSEN6RJAW6OWEGE6ZKYCRA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.460916682 (102h13m40.012387133s) [noise=true , address=5.231.73.2:4018 , piece_id=KP2VQBGZYKUDBJNAIKXV6VRHPK6V3UI6X3SD6HIJOHDTZ6Z6EGLQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.497737346 (102h13m40.049207797s) [noise=true , address=5.231.73.5:4019 , piece_id=H6CZGSTCCYMOZVTSJ2Y5XAKGZA26EQ7E2NONL6DWZGC7Q2LORHBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.511205242 (102h13m40.062675703s) [noise=true , address=5.231.73.3:4030 , piece_id=O4B5VTAWE2WNRG2GZT4OYOEGZLDNYDTHZOT3WETYA3P4XHQR5AOQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.536492093 (102h13m40.087962565s) [noise=true , address=5.231.73.4:4022 , piece_id=GABEYHU5QIUMMQYAHKVXLY6QFY4AZRXMVPSDJ3EO6WOIKA7E4U4Q, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.553999952 (102h13m40.105470433s) [noise=true , address=64.49.13.138:4492 , piece_id=CXNCL6OGRP6C4BXBSH7URFSYJELZLM62KVHGCDJQPD7FVTB67Y3A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.668337736 (102h13m40.219808187s) [noise=true , address=45.91.116.7:4016 , piece_id=BXDTJNCLTSVXQUCC4RO2C2JHZ26XHIJPTHITUO7TXQY6CAY6AC2A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.674376285 (102h13m40.225846746s) [noise=true , address=77.90.31.17:4776 , piece_id=254SH54AOFXNCTTM6QF3AUN7DYYRE2EKBHDBEUKLHEJMKDBXSVDQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.678052054 (102h13m40.229522505s) [noise=true , address=145.79.227.38:4984 , piece_id=LP425WCAJOPGTKLRDOU6V2M3LWRGZ3DZLF4JL4L5QDHW4V37TOZA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.761077612 (102h13m40.312548063s) [noise=true , address=77.90.31.14:4678 , piece_id=D4HEZCFBVRL5CMXNWRCWXFOA3UMT2NXCMIA2L4ACWQ25O3PU4B4A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.771558030 (102h13m40.323028481s) [noise=true , address=45.91.116.6:4993 , piece_id=B7J3JA5UFDKUW6Q6YACXNQY7EUH7PXYRFUOMMLQKOCWJ6NZ3G3DQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.772042953 (102h13m40.323513414s) [noise=true , address=45.91.116.11:4145 , piece_id=7SLQGSEAL7WCGUQ6TRYWNM6F57TZKH6PNDNVXWQGTZ6LKKVXBONA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.775497421 (102h13m40.326967872s) [noise=true , address=45.91.116.14:4203 , piece_id=SU2IO7NSAAQP4ZEY4QIHXQYOHFFDOSGFQEHFJG44QLIPA37DUA6Q, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.775520495 (102h13m40.326990956s) [seg_pos={0 0}, upload=89995]: Segment reached optimal threshold of 14 pieces. Nov 13 14:17:41.775700267 (102h13m40.327170729s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.775707361 (102h13m40.327177812s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.775974550 (102h13m40.327445001s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.775982485 (102h13m40.327452936s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776090991 (102h13m40.327561442s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.776096271 (102h13m40.327566732s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776208986 (102h13m40.327679437s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.776214626 (102h13m40.327685087s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776231869 (102h13m40.32770232s) [seg_pos={0 0}, upload=89995]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 13 14:17:41.776317181 (102h13m40.327787632s) [upload=89995]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:17:41.776322282 (102h13m40.327792743s) [upload=89995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:18:48.053973011 (102h14m46.605443472s) [upload=89995]: Done uploading segments... Nov 13 14:18:48.054067461 (102h14m46.605537912s) [upload=89995]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.29381141s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X.X.........................................] duration: 6.138701985s trace: https://jaeger.dev.storj.io/trace/2a65ce5005cafd1d FAIL (2025-11-13 14:47:49 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.138733295s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.10421441s trace: https://jaeger.dev.storj.io/trace/afd242eeed2927 FAIL (2025-11-13 14:47:40 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.104247081s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.231286611s trace: https://jaeger.dev.storj.io/trace/d6ec5f1d62f8624 FAIL (2025-11-13 14:47:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:32.932592910 (102h12m31.484063371s) [upload=89979]: Starting upload Nov 13 14:16:32.932874286 (102h12m31.484344767s) [upload=89979]: Done starting upload Nov 13 14:16:32.933042266 (102h12m31.484512717s) [upload=89979]: Uploading segments... Nov 13 14:16:32.933200005 (102h12m31.484670467s) [upload=89979]: Got next segment. Inline: true Nov 13 14:16:32.933208482 (102h12m31.484678943s) [upload=89979]: Waiting for error group managing segments... Nov 13 14:16:32.933309854 (102h12m31.484780316s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:32.933317860 (102h12m31.484788322s) [upload=89979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 13 14:16:32.933321948 (102h12m31.484792409s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:17:01.188559179 (102h12m59.74002964s) [upload=89979]: Done uploading segments... Nov 13 14:17:01.188737939 (102h12m59.7402084s) [upload=89979]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.231311288s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 7.461385216s trace: https://jaeger.dev.storj.io/trace/403ba97a5a1cde7d FAIL (2025-11-13 14:47:28 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:00.282283372 (102h11m58.833753833s) [upload=89970]: Starting upload Nov 13 14:16:00.282383773 (102h11m58.833854234s) [upload=89970]: Done starting upload Nov 13 14:16:00.282466601 (102h11m58.833937052s) [upload=89970]: Uploading segments... Nov 13 14:16:00.282681841 (102h11m58.834152302s) [upload=89970]: Got next segment. Inline: false Nov 13 14:16:00.282731475 (102h11m58.834201936s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:00.282737627 (102h11m58.834208088s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 14:16:02.313071205 (102h12m0.864541666s) [seg_pos={0 0}, upload=89970]: Begin upload segment... Nov 13 14:16:02.313149905 (102h12m0.864620376s) [seg_pos={0 0}, upload=89970]: Done begin upload segment. Nov 13 14:16:02.313271697 (102h12m0.864742158s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.313324648 (102h12m0.864795129s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314320904 (102h12m0.865791355s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314744151 (102h12m0.866214642s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315158950 (102h12m0.866629421s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315599659 (102h12m0.867070121s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315928506 (102h12m0.867398967s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.316477040 (102h12m0.867947501s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317049180 (102h12m0.868519651s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317427380 (102h12m0.868897841s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317934746 (102h12m0.869405207s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318376678 (102h12m0.869847139s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318812577 (102h12m0.870283038s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319367303 (102h12m0.870837765s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319957317 (102h12m0.871427768s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.320489491 (102h12m0.871959952s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:03.991369116 (102h12m2.542839567s) [upload=89970]: Next returned nil segment Nov 13 14:16:03.991386198 (102h12m2.542856649s) [upload=89970]: Waiting for error group managing segments... Nov 13 14:16:04.278532114 (102h12m2.830002575s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.279571643 (102h12m2.831042105s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.289684120 (102h12m2.841154581s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.300556964 (102h12m2.852027435s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.301333473 (102h12m2.852803934s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302408349 (102h12m2.853878821s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302631235 (102h12m2.854101716s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.335556962 (102h12m2.887027413s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.346910641 (102h12m2.898381102s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.371810816 (102h12m2.923281277s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377236428 (102h12m2.928706889s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377262828 (102h12m2.928733289s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383693804 (102h12m2.935164265s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383710837 (102h12m2.935181288s) [seg_pos={0 0}, upload=89970]: Segment reached optimal threshold of 13 pieces. Nov 13 14:16:04.383822670 (102h12m2.935293121s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383922450 (102h12m2.935392911s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ): protocol: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection Nov 13 14:16:04.383929954 (102h12m2.935400415s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384043279 (102h12m2.93551374s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection Nov 13 14:16:04.384050112 (102h12m2.935520563s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384066143 (102h12m2.935536604s) [seg_pos={0 0}, upload=89970]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Nov 13 14:16:04.384159561 (102h12m2.935630013s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:16:04.384171734 (102h12m2.935642195s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:16:04.934815134 (102h12m3.486285595s) [upload=89970]: Done uploading segments... Nov 13 14:16:04.934864327 (102h12m3.486334788s) [upload=89970]: Upload finished. err: <nil> Duration: 7.461417528s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X.X........................................] duration: 6.248931537s trace: https://jaeger.dev.storj.io/trace/2a65ce5005cafd1d FAIL (2025-11-13 14:47:12 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.248981672s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.178973041s trace: https://jaeger.dev.storj.io/trace/d6ec5f1d62f8624 FAIL (2025-11-13 14:47:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:32.932592910 (102h12m31.484063371s) [upload=89979]: Starting upload Nov 13 14:16:32.932874286 (102h12m31.484344767s) [upload=89979]: Done starting upload Nov 13 14:16:32.933042266 (102h12m31.484512717s) [upload=89979]: Uploading segments... Nov 13 14:16:32.933200005 (102h12m31.484670467s) [upload=89979]: Got next segment. Inline: true Nov 13 14:16:32.933208482 (102h12m31.484678943s) [upload=89979]: Waiting for error group managing segments... Nov 13 14:16:32.933309854 (102h12m31.484780316s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:32.933317860 (102h12m31.484788322s) [upload=89979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 13 14:16:32.933321948 (102h12m31.484792409s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:17:01.188559179 (102h12m59.74002964s) [upload=89979]: Done uploading segments... Nov 13 14:17:01.188737939 (102h12m59.7402084s) [upload=89979]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.179000032s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.323054679s trace: https://jaeger.dev.storj.io/trace/403ba97a5a1cde7d FAIL (2025-11-13 14:46:56 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:00.282283372 (102h11m58.833753833s) [upload=89970]: Starting upload Nov 13 14:16:00.282383773 (102h11m58.833854234s) [upload=89970]: Done starting upload Nov 13 14:16:00.282466601 (102h11m58.833937052s) [upload=89970]: Uploading segments... Nov 13 14:16:00.282681841 (102h11m58.834152302s) [upload=89970]: Got next segment. Inline: false Nov 13 14:16:00.282731475 (102h11m58.834201936s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:00.282737627 (102h11m58.834208088s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 14:16:02.313071205 (102h12m0.864541666s) [seg_pos={0 0}, upload=89970]: Begin upload segment... Nov 13 14:16:02.313149905 (102h12m0.864620376s) [seg_pos={0 0}, upload=89970]: Done begin upload segment. Nov 13 14:16:02.313271697 (102h12m0.864742158s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.313324648 (102h12m0.864795129s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314320904 (102h12m0.865791355s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314744151 (102h12m0.866214642s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315158950 (102h12m0.866629421s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315599659 (102h12m0.867070121s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315928506 (102h12m0.867398967s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.316477040 (102h12m0.867947501s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317049180 (102h12m0.868519651s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317427380 (102h12m0.868897841s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317934746 (102h12m0.869405207s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318376678 (102h12m0.869847139s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318812577 (102h12m0.870283038s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319367303 (102h12m0.870837765s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319957317 (102h12m0.871427768s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.320489491 (102h12m0.871959952s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:03.991369116 (102h12m2.542839567s) [upload=89970]: Next returned nil segment Nov 13 14:16:03.991386198 (102h12m2.542856649s) [upload=89970]: Waiting for error group managing segments... Nov 13 14:16:04.278532114 (102h12m2.830002575s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.279571643 (102h12m2.831042105s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.289684120 (102h12m2.841154581s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.300556964 (102h12m2.852027435s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.301333473 (102h12m2.852803934s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302408349 (102h12m2.853878821s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302631235 (102h12m2.854101716s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.335556962 (102h12m2.887027413s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.346910641 (102h12m2.898381102s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.371810816 (102h12m2.923281277s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377236428 (102h12m2.928706889s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377262828 (102h12m2.928733289s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383693804 (102h12m2.935164265s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383710837 (102h12m2.935181288s) [seg_pos={0 0}, upload=89970]: Segment reached optimal threshold of 13 pieces. Nov 13 14:16:04.383822670 (102h12m2.935293121s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383922450 (102h12m2.935392911s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ): protocol: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection Nov 13 14:16:04.383929954 (102h12m2.935400415s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384043279 (102h12m2.93551374s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection Nov 13 14:16:04.384050112 (102h12m2.935520563s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384066143 (102h12m2.935536604s) [seg_pos={0 0}, upload=89970]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Nov 13 14:16:04.384159561 (102h12m2.935630013s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:16:04.384171734 (102h12m2.935642195s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:16:04.934815134 (102h12m3.486285595s) [upload=89970]: Done uploading segments... Nov 13 14:16:04.934864327 (102h12m3.486334788s) [upload=89970]: Upload finished. err: <nil> Duration: 6.323082683s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.195062885s trace: https://jaeger.dev.storj.io/trace/d53ad46a3bad038 FAIL (2025-11-13 14:46:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ... Nov 13 14:17:39.718806610 (102h13m38.270277071s) [noise=true , address=5.231.73.3:4030 , piece_id=O4B5VTAWE2WNRG2GZT4OYOEGZLDNYDTHZOT3WETYA3P4XHQR5AOQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.718869189 (102h13m38.27033964s) [noise=true , address=45.91.116.7:4016 , piece_id=BXDTJNCLTSVXQUCC4RO2C2JHZ26XHIJPTHITUO7TXQY6CAY6AC2A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719243782 (102h13m38.270714243s) [noise=true , address=145.79.227.38:4984 , piece_id=LP425WCAJOPGTKLRDOU6V2M3LWRGZ3DZLF4JL4L5QDHW4V37TOZA, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719600372 (102h13m38.271070833s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719757630 (102h13m38.271228102s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719982179 (102h13m38.27145265s) [noise=true , address=5.231.73.2:4018 , piece_id=KP2VQBGZYKUDBJNAIKXV6VRHPK6V3UI6X3SD6HIJOHDTZ6Z6EGLQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720312697 (102h13m38.271783169s) [noise=true , address=77.90.31.17:4776 , piece_id=254SH54AOFXNCTTM6QF3AUN7DYYRE2EKBHDBEUKLHEJMKDBXSVDQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720732096 (102h13m38.272202567s) [noise=true , address=45.91.116.6:4993 , piece_id=B7J3JA5UFDKUW6Q6YACXNQY7EUH7PXYRFUOMMLQKOCWJ6NZ3G3DQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720836485 (102h13m38.272306946s) [noise=true , address=45.91.116.11:4145 , piece_id=7SLQGSEAL7WCGUQ6TRYWNM6F57TZKH6PNDNVXWQGTZ6LKKVXBONA, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.721147126 (102h13m38.272617587s) [noise=true , address=77.90.31.14:4678 , piece_id=D4HEZCFBVRL5CMXNWRCWXFOA3UMT2NXCMIA2L4ACWQ25O3PU4B4A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.721537239 (102h13m38.2730077s) [noise=true , address=5.231.73.4:4022 , piece_id=GABEYHU5QIUMMQYAHKVXLY6QFY4AZRXMVPSDJ3EO6WOIKA7E4U4Q, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.722351200 (102h13m38.273821661s) [noise=true , address=45.91.116.14:4203 , piece_id=SU2IO7NSAAQP4ZEY4QIHXQYOHFFDOSGFQEHFJG44QLIPA37DUA6Q, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.723267113 (102h13m38.274737584s) [noise=true , address=5.231.73.5:4019 , piece_id=H6CZGSTCCYMOZVTSJ2Y5XAKGZA26EQ7E2NONL6DWZGC7Q2LORHBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.723886824 (102h13m38.275357285s) [noise=true , address=64.49.13.26:4948 , piece_id=MQ6RQQQ36YHFSTJYDE44AATC7LQPIFCB7QOS72DMTWRETZVAQWDQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.724315690 (102h13m38.275786151s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.724792528 (102h13m38.276262989s) [noise=true , address=64.49.13.138:4492 , piece_id=CXNCL6OGRP6C4BXBSH7URFSYJELZLM62KVHGCDJQPD7FVTB67Y3A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.725327326 (102h13m38.276797787s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:41.027262394 (102h13m39.578732845s) [upload=89995]: Next returned nil segment Nov 13 14:17:41.027282332 (102h13m39.578752783s) [upload=89995]: Waiting for error group managing segments... Nov 13 14:17:41.357905237 (102h13m39.909375698s) [noise=true , address=64.49.13.26:4948 , piece_id=MQ6RQQQ36YHFSTJYDE44AATC7LQPIFCB7QOS72DMTWRETZVAQWDQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.433145731 (102h13m39.984616192s) [noise=true , address=64.49.13.154:4929 , piece_id=2K55IIOVKBDVMEO3QKXXOZ65X35SQIVSEN6RJAW6OWEGE6ZKYCRA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.460916682 (102h13m40.012387133s) [noise=true , address=5.231.73.2:4018 , piece_id=KP2VQBGZYKUDBJNAIKXV6VRHPK6V3UI6X3SD6HIJOHDTZ6Z6EGLQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.497737346 (102h13m40.049207797s) [noise=true , address=5.231.73.5:4019 , piece_id=H6CZGSTCCYMOZVTSJ2Y5XAKGZA26EQ7E2NONL6DWZGC7Q2LORHBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.511205242 (102h13m40.062675703s) [noise=true , address=5.231.73.3:4030 , piece_id=O4B5VTAWE2WNRG2GZT4OYOEGZLDNYDTHZOT3WETYA3P4XHQR5AOQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.536492093 (102h13m40.087962565s) [noise=true , address=5.231.73.4:4022 , piece_id=GABEYHU5QIUMMQYAHKVXLY6QFY4AZRXMVPSDJ3EO6WOIKA7E4U4Q, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.553999952 (102h13m40.105470433s) [noise=true , address=64.49.13.138:4492 , piece_id=CXNCL6OGRP6C4BXBSH7URFSYJELZLM62KVHGCDJQPD7FVTB67Y3A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.668337736 (102h13m40.219808187s) [noise=true , address=45.91.116.7:4016 , piece_id=BXDTJNCLTSVXQUCC4RO2C2JHZ26XHIJPTHITUO7TXQY6CAY6AC2A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.674376285 (102h13m40.225846746s) [noise=true , address=77.90.31.17:4776 , piece_id=254SH54AOFXNCTTM6QF3AUN7DYYRE2EKBHDBEUKLHEJMKDBXSVDQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.678052054 (102h13m40.229522505s) [noise=true , address=145.79.227.38:4984 , piece_id=LP425WCAJOPGTKLRDOU6V2M3LWRGZ3DZLF4JL4L5QDHW4V37TOZA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.761077612 (102h13m40.312548063s) [noise=true , address=77.90.31.14:4678 , piece_id=D4HEZCFBVRL5CMXNWRCWXFOA3UMT2NXCMIA2L4ACWQ25O3PU4B4A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.771558030 (102h13m40.323028481s) [noise=true , address=45.91.116.6:4993 , piece_id=B7J3JA5UFDKUW6Q6YACXNQY7EUH7PXYRFUOMMLQKOCWJ6NZ3G3DQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.772042953 (102h13m40.323513414s) [noise=true , address=45.91.116.11:4145 , piece_id=7SLQGSEAL7WCGUQ6TRYWNM6F57TZKH6PNDNVXWQGTZ6LKKVXBONA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.775497421 (102h13m40.326967872s) [noise=true , address=45.91.116.14:4203 , piece_id=SU2IO7NSAAQP4ZEY4QIHXQYOHFFDOSGFQEHFJG44QLIPA37DUA6Q, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.775520495 (102h13m40.326990956s) [seg_pos={0 0}, upload=89995]: Segment reached optimal threshold of 14 pieces. Nov 13 14:17:41.775700267 (102h13m40.327170729s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.775707361 (102h13m40.327177812s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.775974550 (102h13m40.327445001s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.775982485 (102h13m40.327452936s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776090991 (102h13m40.327561442s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.776096271 (102h13m40.327566732s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776208986 (102h13m40.327679437s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.776214626 (102h13m40.327685087s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776231869 (102h13m40.32770232s) [seg_pos={0 0}, upload=89995]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 13 14:17:41.776317181 (102h13m40.327787632s) [upload=89995]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:17:41.776322282 (102h13m40.327792743s) [upload=89995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:18:48.053973011 (102h14m46.605443472s) [upload=89995]: Done uploading segments... Nov 13 14:18:48.054067461 (102h14m46.605537912s) [upload=89995]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.19509185s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.119432105s trace: https://jaeger.dev.storj.io/trace/afd242eeed2927 FAIL (2025-11-13 14:46:25 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.119466239s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.134317391s trace: https://jaeger.dev.storj.io/trace/d6ec5f1d62f8624 FAIL (2025-11-13 14:46:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:32.932592910 (102h12m31.484063371s) [upload=89979]: Starting upload Nov 13 14:16:32.932874286 (102h12m31.484344767s) [upload=89979]: Done starting upload Nov 13 14:16:32.933042266 (102h12m31.484512717s) [upload=89979]: Uploading segments... Nov 13 14:16:32.933200005 (102h12m31.484670467s) [upload=89979]: Got next segment. Inline: true Nov 13 14:16:32.933208482 (102h12m31.484678943s) [upload=89979]: Waiting for error group managing segments... Nov 13 14:16:32.933309854 (102h12m31.484780316s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:32.933317860 (102h12m31.484788322s) [upload=89979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 13 14:16:32.933321948 (102h12m31.484792409s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:17:01.188559179 (102h12m59.74002964s) [upload=89979]: Done uploading segments... Nov 13 14:17:01.188737939 (102h12m59.7402084s) [upload=89979]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.134347967s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.658488584s trace: https://jaeger.dev.storj.io/trace/403ba97a5a1cde7d FAIL (2025-11-13 14:46:07 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:00.282283372 (102h11m58.833753833s) [upload=89970]: Starting upload Nov 13 14:16:00.282383773 (102h11m58.833854234s) [upload=89970]: Done starting upload Nov 13 14:16:00.282466601 (102h11m58.833937052s) [upload=89970]: Uploading segments... Nov 13 14:16:00.282681841 (102h11m58.834152302s) [upload=89970]: Got next segment. Inline: false Nov 13 14:16:00.282731475 (102h11m58.834201936s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:00.282737627 (102h11m58.834208088s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 14:16:02.313071205 (102h12m0.864541666s) [seg_pos={0 0}, upload=89970]: Begin upload segment... Nov 13 14:16:02.313149905 (102h12m0.864620376s) [seg_pos={0 0}, upload=89970]: Done begin upload segment. Nov 13 14:16:02.313271697 (102h12m0.864742158s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.313324648 (102h12m0.864795129s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314320904 (102h12m0.865791355s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314744151 (102h12m0.866214642s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315158950 (102h12m0.866629421s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315599659 (102h12m0.867070121s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315928506 (102h12m0.867398967s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.316477040 (102h12m0.867947501s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317049180 (102h12m0.868519651s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317427380 (102h12m0.868897841s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317934746 (102h12m0.869405207s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318376678 (102h12m0.869847139s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318812577 (102h12m0.870283038s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319367303 (102h12m0.870837765s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319957317 (102h12m0.871427768s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.320489491 (102h12m0.871959952s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:03.991369116 (102h12m2.542839567s) [upload=89970]: Next returned nil segment Nov 13 14:16:03.991386198 (102h12m2.542856649s) [upload=89970]: Waiting for error group managing segments... Nov 13 14:16:04.278532114 (102h12m2.830002575s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.279571643 (102h12m2.831042105s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.289684120 (102h12m2.841154581s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.300556964 (102h12m2.852027435s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.301333473 (102h12m2.852803934s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302408349 (102h12m2.853878821s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302631235 (102h12m2.854101716s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.335556962 (102h12m2.887027413s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.346910641 (102h12m2.898381102s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.371810816 (102h12m2.923281277s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377236428 (102h12m2.928706889s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377262828 (102h12m2.928733289s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383693804 (102h12m2.935164265s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383710837 (102h12m2.935181288s) [seg_pos={0 0}, upload=89970]: Segment reached optimal threshold of 13 pieces. Nov 13 14:16:04.383822670 (102h12m2.935293121s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383922450 (102h12m2.935392911s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ): protocol: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection Nov 13 14:16:04.383929954 (102h12m2.935400415s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384043279 (102h12m2.93551374s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection Nov 13 14:16:04.384050112 (102h12m2.935520563s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384066143 (102h12m2.935536604s) [seg_pos={0 0}, upload=89970]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Nov 13 14:16:04.384159561 (102h12m2.935630013s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:16:04.384171734 (102h12m2.935642195s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:16:04.934815134 (102h12m3.486285595s) [upload=89970]: Done uploading segments... Nov 13 14:16:04.934864327 (102h12m3.486334788s) [upload=89970]: Upload finished. err: <nil> Duration: 6.658521787s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.940054426s trace: https://jaeger.dev.storj.io/trace/d53ad46a3bad038 FAIL (2025-11-13 14:46:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ... Nov 13 14:17:39.718806610 (102h13m38.270277071s) [noise=true , address=5.231.73.3:4030 , piece_id=O4B5VTAWE2WNRG2GZT4OYOEGZLDNYDTHZOT3WETYA3P4XHQR5AOQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.718869189 (102h13m38.27033964s) [noise=true , address=45.91.116.7:4016 , piece_id=BXDTJNCLTSVXQUCC4RO2C2JHZ26XHIJPTHITUO7TXQY6CAY6AC2A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719243782 (102h13m38.270714243s) [noise=true , address=145.79.227.38:4984 , piece_id=LP425WCAJOPGTKLRDOU6V2M3LWRGZ3DZLF4JL4L5QDHW4V37TOZA, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719600372 (102h13m38.271070833s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719757630 (102h13m38.271228102s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719982179 (102h13m38.27145265s) [noise=true , address=5.231.73.2:4018 , piece_id=KP2VQBGZYKUDBJNAIKXV6VRHPK6V3UI6X3SD6HIJOHDTZ6Z6EGLQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720312697 (102h13m38.271783169s) [noise=true , address=77.90.31.17:4776 , piece_id=254SH54AOFXNCTTM6QF3AUN7DYYRE2EKBHDBEUKLHEJMKDBXSVDQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720732096 (102h13m38.272202567s) [noise=true , address=45.91.116.6:4993 , piece_id=B7J3JA5UFDKUW6Q6YACXNQY7EUH7PXYRFUOMMLQKOCWJ6NZ3G3DQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720836485 (102h13m38.272306946s) [noise=true , address=45.91.116.11:4145 , piece_id=7SLQGSEAL7WCGUQ6TRYWNM6F57TZKH6PNDNVXWQGTZ6LKKVXBONA, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.721147126 (102h13m38.272617587s) [noise=true , address=77.90.31.14:4678 , piece_id=D4HEZCFBVRL5CMXNWRCWXFOA3UMT2NXCMIA2L4ACWQ25O3PU4B4A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.721537239 (102h13m38.2730077s) [noise=true , address=5.231.73.4:4022 , piece_id=GABEYHU5QIUMMQYAHKVXLY6QFY4AZRXMVPSDJ3EO6WOIKA7E4U4Q, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.722351200 (102h13m38.273821661s) [noise=true , address=45.91.116.14:4203 , piece_id=SU2IO7NSAAQP4ZEY4QIHXQYOHFFDOSGFQEHFJG44QLIPA37DUA6Q, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.723267113 (102h13m38.274737584s) [noise=true , address=5.231.73.5:4019 , piece_id=H6CZGSTCCYMOZVTSJ2Y5XAKGZA26EQ7E2NONL6DWZGC7Q2LORHBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.723886824 (102h13m38.275357285s) [noise=true , address=64.49.13.26:4948 , piece_id=MQ6RQQQ36YHFSTJYDE44AATC7LQPIFCB7QOS72DMTWRETZVAQWDQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.724315690 (102h13m38.275786151s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.724792528 (102h13m38.276262989s) [noise=true , address=64.49.13.138:4492 , piece_id=CXNCL6OGRP6C4BXBSH7URFSYJELZLM62KVHGCDJQPD7FVTB67Y3A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.725327326 (102h13m38.276797787s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:41.027262394 (102h13m39.578732845s) [upload=89995]: Next returned nil segment Nov 13 14:17:41.027282332 (102h13m39.578752783s) [upload=89995]: Waiting for error group managing segments... Nov 13 14:17:41.357905237 (102h13m39.909375698s) [noise=true , address=64.49.13.26:4948 , piece_id=MQ6RQQQ36YHFSTJYDE44AATC7LQPIFCB7QOS72DMTWRETZVAQWDQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.433145731 (102h13m39.984616192s) [noise=true , address=64.49.13.154:4929 , piece_id=2K55IIOVKBDVMEO3QKXXOZ65X35SQIVSEN6RJAW6OWEGE6ZKYCRA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.460916682 (102h13m40.012387133s) [noise=true , address=5.231.73.2:4018 , piece_id=KP2VQBGZYKUDBJNAIKXV6VRHPK6V3UI6X3SD6HIJOHDTZ6Z6EGLQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.497737346 (102h13m40.049207797s) [noise=true , address=5.231.73.5:4019 , piece_id=H6CZGSTCCYMOZVTSJ2Y5XAKGZA26EQ7E2NONL6DWZGC7Q2LORHBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.511205242 (102h13m40.062675703s) [noise=true , address=5.231.73.3:4030 , piece_id=O4B5VTAWE2WNRG2GZT4OYOEGZLDNYDTHZOT3WETYA3P4XHQR5AOQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.536492093 (102h13m40.087962565s) [noise=true , address=5.231.73.4:4022 , piece_id=GABEYHU5QIUMMQYAHKVXLY6QFY4AZRXMVPSDJ3EO6WOIKA7E4U4Q, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.553999952 (102h13m40.105470433s) [noise=true , address=64.49.13.138:4492 , piece_id=CXNCL6OGRP6C4BXBSH7URFSYJELZLM62KVHGCDJQPD7FVTB67Y3A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.668337736 (102h13m40.219808187s) [noise=true , address=45.91.116.7:4016 , piece_id=BXDTJNCLTSVXQUCC4RO2C2JHZ26XHIJPTHITUO7TXQY6CAY6AC2A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.674376285 (102h13m40.225846746s) [noise=true , address=77.90.31.17:4776 , piece_id=254SH54AOFXNCTTM6QF3AUN7DYYRE2EKBHDBEUKLHEJMKDBXSVDQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.678052054 (102h13m40.229522505s) [noise=true , address=145.79.227.38:4984 , piece_id=LP425WCAJOPGTKLRDOU6V2M3LWRGZ3DZLF4JL4L5QDHW4V37TOZA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.761077612 (102h13m40.312548063s) [noise=true , address=77.90.31.14:4678 , piece_id=D4HEZCFBVRL5CMXNWRCWXFOA3UMT2NXCMIA2L4ACWQ25O3PU4B4A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.771558030 (102h13m40.323028481s) [noise=true , address=45.91.116.6:4993 , piece_id=B7J3JA5UFDKUW6Q6YACXNQY7EUH7PXYRFUOMMLQKOCWJ6NZ3G3DQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.772042953 (102h13m40.323513414s) [noise=true , address=45.91.116.11:4145 , piece_id=7SLQGSEAL7WCGUQ6TRYWNM6F57TZKH6PNDNVXWQGTZ6LKKVXBONA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.775497421 (102h13m40.326967872s) [noise=true , address=45.91.116.14:4203 , piece_id=SU2IO7NSAAQP4ZEY4QIHXQYOHFFDOSGFQEHFJG44QLIPA37DUA6Q, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.775520495 (102h13m40.326990956s) [seg_pos={0 0}, upload=89995]: Segment reached optimal threshold of 14 pieces. Nov 13 14:17:41.775700267 (102h13m40.327170729s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.775707361 (102h13m40.327177812s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.775974550 (102h13m40.327445001s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.775982485 (102h13m40.327452936s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776090991 (102h13m40.327561442s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.776096271 (102h13m40.327566732s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776208986 (102h13m40.327679437s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.776214626 (102h13m40.327685087s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776231869 (102h13m40.32770232s) [seg_pos={0 0}, upload=89995]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 13 14:17:41.776317181 (102h13m40.327787632s) [upload=89995]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:17:41.776322282 (102h13m40.327792743s) [upload=89995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:18:48.053973011 (102h14m46.605443472s) [upload=89995]: Done uploading segments... Nov 13 14:18:48.054067461 (102h14m46.605537912s) [upload=89995]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.940095234s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X.X.......................................] duration: 6.192277039s trace: https://jaeger.dev.storj.io/trace/2a65ce5005cafd1d FAIL (2025-11-13 14:45:59 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.192306274s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.504458706s trace: https://jaeger.dev.storj.io/trace/d6ec5f1d62f8624 FAIL (2025-11-13 14:45:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:32.932592910 (102h12m31.484063371s) [upload=89979]: Starting upload Nov 13 14:16:32.932874286 (102h12m31.484344767s) [upload=89979]: Done starting upload Nov 13 14:16:32.933042266 (102h12m31.484512717s) [upload=89979]: Uploading segments... Nov 13 14:16:32.933200005 (102h12m31.484670467s) [upload=89979]: Got next segment. Inline: true Nov 13 14:16:32.933208482 (102h12m31.484678943s) [upload=89979]: Waiting for error group managing segments... Nov 13 14:16:32.933309854 (102h12m31.484780316s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:32.933317860 (102h12m31.484788322s) [upload=89979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 13 14:16:32.933321948 (102h12m31.484792409s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:17:01.188559179 (102h12m59.74002964s) [upload=89979]: Done uploading segments... Nov 13 14:17:01.188737939 (102h12m59.7402084s) [upload=89979]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.504488405s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X.X......................................] duration: 6.040881534s trace: https://jaeger.dev.storj.io/trace/2a65ce5005cafd1d FAIL (2025-11-13 14:45:38 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.040946898s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.546830863s trace: https://jaeger.dev.storj.io/trace/403ba97a5a1cde7d FAIL (2025-11-13 14:45:22 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:00.282283372 (102h11m58.833753833s) [upload=89970]: Starting upload Nov 13 14:16:00.282383773 (102h11m58.833854234s) [upload=89970]: Done starting upload Nov 13 14:16:00.282466601 (102h11m58.833937052s) [upload=89970]: Uploading segments... Nov 13 14:16:00.282681841 (102h11m58.834152302s) [upload=89970]: Got next segment. Inline: false Nov 13 14:16:00.282731475 (102h11m58.834201936s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:00.282737627 (102h11m58.834208088s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 14:16:02.313071205 (102h12m0.864541666s) [seg_pos={0 0}, upload=89970]: Begin upload segment... Nov 13 14:16:02.313149905 (102h12m0.864620376s) [seg_pos={0 0}, upload=89970]: Done begin upload segment. Nov 13 14:16:02.313271697 (102h12m0.864742158s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.313324648 (102h12m0.864795129s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314320904 (102h12m0.865791355s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314744151 (102h12m0.866214642s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315158950 (102h12m0.866629421s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315599659 (102h12m0.867070121s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315928506 (102h12m0.867398967s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.316477040 (102h12m0.867947501s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317049180 (102h12m0.868519651s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317427380 (102h12m0.868897841s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317934746 (102h12m0.869405207s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318376678 (102h12m0.869847139s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318812577 (102h12m0.870283038s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319367303 (102h12m0.870837765s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319957317 (102h12m0.871427768s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.320489491 (102h12m0.871959952s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:03.991369116 (102h12m2.542839567s) [upload=89970]: Next returned nil segment Nov 13 14:16:03.991386198 (102h12m2.542856649s) [upload=89970]: Waiting for error group managing segments... Nov 13 14:16:04.278532114 (102h12m2.830002575s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.279571643 (102h12m2.831042105s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.289684120 (102h12m2.841154581s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.300556964 (102h12m2.852027435s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.301333473 (102h12m2.852803934s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302408349 (102h12m2.853878821s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302631235 (102h12m2.854101716s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.335556962 (102h12m2.887027413s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.346910641 (102h12m2.898381102s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.371810816 (102h12m2.923281277s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377236428 (102h12m2.928706889s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377262828 (102h12m2.928733289s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383693804 (102h12m2.935164265s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383710837 (102h12m2.935181288s) [seg_pos={0 0}, upload=89970]: Segment reached optimal threshold of 13 pieces. Nov 13 14:16:04.383822670 (102h12m2.935293121s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383922450 (102h12m2.935392911s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ): protocol: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection Nov 13 14:16:04.383929954 (102h12m2.935400415s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384043279 (102h12m2.93551374s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection Nov 13 14:16:04.384050112 (102h12m2.935520563s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384066143 (102h12m2.935536604s) [seg_pos={0 0}, upload=89970]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Nov 13 14:16:04.384159561 (102h12m2.935630013s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:16:04.384171734 (102h12m2.935642195s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:16:04.934815134 (102h12m3.486285595s) [upload=89970]: Done uploading segments... Nov 13 14:16:04.934864327 (102h12m3.486334788s) [upload=89970]: Upload finished. err: <nil> Duration: 6.546857845s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.106807283s trace: https://jaeger.dev.storj.io/trace/afd242eeed2927 FAIL (2025-11-13 14:45:22 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.106839104s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.221741852s trace: https://jaeger.dev.storj.io/trace/d6ec5f1d62f8624 FAIL (2025-11-13 14:45:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:32.932592910 (102h12m31.484063371s) [upload=89979]: Starting upload Nov 13 14:16:32.932874286 (102h12m31.484344767s) [upload=89979]: Done starting upload Nov 13 14:16:32.933042266 (102h12m31.484512717s) [upload=89979]: Uploading segments... Nov 13 14:16:32.933200005 (102h12m31.484670467s) [upload=89979]: Got next segment. Inline: true Nov 13 14:16:32.933208482 (102h12m31.484678943s) [upload=89979]: Waiting for error group managing segments... Nov 13 14:16:32.933309854 (102h12m31.484780316s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:32.933317860 (102h12m31.484788322s) [upload=89979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 13 14:16:32.933321948 (102h12m31.484792409s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:17:01.188559179 (102h12m59.74002964s) [upload=89979]: Done uploading segments... Nov 13 14:17:01.188737939 (102h12m59.7402084s) [upload=89979]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.22177183s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.850810954s trace: https://jaeger.dev.storj.io/trace/d53ad46a3bad038 FAIL (2025-11-13 14:44:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ... Nov 13 14:17:39.718806610 (102h13m38.270277071s) [noise=true , address=5.231.73.3:4030 , piece_id=O4B5VTAWE2WNRG2GZT4OYOEGZLDNYDTHZOT3WETYA3P4XHQR5AOQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.718869189 (102h13m38.27033964s) [noise=true , address=45.91.116.7:4016 , piece_id=BXDTJNCLTSVXQUCC4RO2C2JHZ26XHIJPTHITUO7TXQY6CAY6AC2A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719243782 (102h13m38.270714243s) [noise=true , address=145.79.227.38:4984 , piece_id=LP425WCAJOPGTKLRDOU6V2M3LWRGZ3DZLF4JL4L5QDHW4V37TOZA, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719600372 (102h13m38.271070833s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719757630 (102h13m38.271228102s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719982179 (102h13m38.27145265s) [noise=true , address=5.231.73.2:4018 , piece_id=KP2VQBGZYKUDBJNAIKXV6VRHPK6V3UI6X3SD6HIJOHDTZ6Z6EGLQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720312697 (102h13m38.271783169s) [noise=true , address=77.90.31.17:4776 , piece_id=254SH54AOFXNCTTM6QF3AUN7DYYRE2EKBHDBEUKLHEJMKDBXSVDQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720732096 (102h13m38.272202567s) [noise=true , address=45.91.116.6:4993 , piece_id=B7J3JA5UFDKUW6Q6YACXNQY7EUH7PXYRFUOMMLQKOCWJ6NZ3G3DQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720836485 (102h13m38.272306946s) [noise=true , address=45.91.116.11:4145 , piece_id=7SLQGSEAL7WCGUQ6TRYWNM6F57TZKH6PNDNVXWQGTZ6LKKVXBONA, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.721147126 (102h13m38.272617587s) [noise=true , address=77.90.31.14:4678 , piece_id=D4HEZCFBVRL5CMXNWRCWXFOA3UMT2NXCMIA2L4ACWQ25O3PU4B4A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.721537239 (102h13m38.2730077s) [noise=true , address=5.231.73.4:4022 , piece_id=GABEYHU5QIUMMQYAHKVXLY6QFY4AZRXMVPSDJ3EO6WOIKA7E4U4Q, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.722351200 (102h13m38.273821661s) [noise=true , address=45.91.116.14:4203 , piece_id=SU2IO7NSAAQP4ZEY4QIHXQYOHFFDOSGFQEHFJG44QLIPA37DUA6Q, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.723267113 (102h13m38.274737584s) [noise=true , address=5.231.73.5:4019 , piece_id=H6CZGSTCCYMOZVTSJ2Y5XAKGZA26EQ7E2NONL6DWZGC7Q2LORHBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.723886824 (102h13m38.275357285s) [noise=true , address=64.49.13.26:4948 , piece_id=MQ6RQQQ36YHFSTJYDE44AATC7LQPIFCB7QOS72DMTWRETZVAQWDQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.724315690 (102h13m38.275786151s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.724792528 (102h13m38.276262989s) [noise=true , address=64.49.13.138:4492 , piece_id=CXNCL6OGRP6C4BXBSH7URFSYJELZLM62KVHGCDJQPD7FVTB67Y3A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.725327326 (102h13m38.276797787s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:41.027262394 (102h13m39.578732845s) [upload=89995]: Next returned nil segment Nov 13 14:17:41.027282332 (102h13m39.578752783s) [upload=89995]: Waiting for error group managing segments... Nov 13 14:17:41.357905237 (102h13m39.909375698s) [noise=true , address=64.49.13.26:4948 , piece_id=MQ6RQQQ36YHFSTJYDE44AATC7LQPIFCB7QOS72DMTWRETZVAQWDQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.433145731 (102h13m39.984616192s) [noise=true , address=64.49.13.154:4929 , piece_id=2K55IIOVKBDVMEO3QKXXOZ65X35SQIVSEN6RJAW6OWEGE6ZKYCRA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.460916682 (102h13m40.012387133s) [noise=true , address=5.231.73.2:4018 , piece_id=KP2VQBGZYKUDBJNAIKXV6VRHPK6V3UI6X3SD6HIJOHDTZ6Z6EGLQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.497737346 (102h13m40.049207797s) [noise=true , address=5.231.73.5:4019 , piece_id=H6CZGSTCCYMOZVTSJ2Y5XAKGZA26EQ7E2NONL6DWZGC7Q2LORHBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.511205242 (102h13m40.062675703s) [noise=true , address=5.231.73.3:4030 , piece_id=O4B5VTAWE2WNRG2GZT4OYOEGZLDNYDTHZOT3WETYA3P4XHQR5AOQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.536492093 (102h13m40.087962565s) [noise=true , address=5.231.73.4:4022 , piece_id=GABEYHU5QIUMMQYAHKVXLY6QFY4AZRXMVPSDJ3EO6WOIKA7E4U4Q, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.553999952 (102h13m40.105470433s) [noise=true , address=64.49.13.138:4492 , piece_id=CXNCL6OGRP6C4BXBSH7URFSYJELZLM62KVHGCDJQPD7FVTB67Y3A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.668337736 (102h13m40.219808187s) [noise=true , address=45.91.116.7:4016 , piece_id=BXDTJNCLTSVXQUCC4RO2C2JHZ26XHIJPTHITUO7TXQY6CAY6AC2A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.674376285 (102h13m40.225846746s) [noise=true , address=77.90.31.17:4776 , piece_id=254SH54AOFXNCTTM6QF3AUN7DYYRE2EKBHDBEUKLHEJMKDBXSVDQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.678052054 (102h13m40.229522505s) [noise=true , address=145.79.227.38:4984 , piece_id=LP425WCAJOPGTKLRDOU6V2M3LWRGZ3DZLF4JL4L5QDHW4V37TOZA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.761077612 (102h13m40.312548063s) [noise=true , address=77.90.31.14:4678 , piece_id=D4HEZCFBVRL5CMXNWRCWXFOA3UMT2NXCMIA2L4ACWQ25O3PU4B4A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.771558030 (102h13m40.323028481s) [noise=true , address=45.91.116.6:4993 , piece_id=B7J3JA5UFDKUW6Q6YACXNQY7EUH7PXYRFUOMMLQKOCWJ6NZ3G3DQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.772042953 (102h13m40.323513414s) [noise=true , address=45.91.116.11:4145 , piece_id=7SLQGSEAL7WCGUQ6TRYWNM6F57TZKH6PNDNVXWQGTZ6LKKVXBONA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.775497421 (102h13m40.326967872s) [noise=true , address=45.91.116.14:4203 , piece_id=SU2IO7NSAAQP4ZEY4QIHXQYOHFFDOSGFQEHFJG44QLIPA37DUA6Q, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.775520495 (102h13m40.326990956s) [seg_pos={0 0}, upload=89995]: Segment reached optimal threshold of 14 pieces. Nov 13 14:17:41.775700267 (102h13m40.327170729s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.775707361 (102h13m40.327177812s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.775974550 (102h13m40.327445001s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.775982485 (102h13m40.327452936s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776090991 (102h13m40.327561442s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.776096271 (102h13m40.327566732s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776208986 (102h13m40.327679437s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.776214626 (102h13m40.327685087s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776231869 (102h13m40.32770232s) [seg_pos={0 0}, upload=89995]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 13 14:17:41.776317181 (102h13m40.327787632s) [upload=89995]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:17:41.776322282 (102h13m40.327792743s) [upload=89995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:18:48.053973011 (102h14m46.605443472s) [upload=89995]: Done uploading segments... Nov 13 14:18:48.054067461 (102h14m46.605537912s) [upload=89995]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.850840841s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.224826048s trace: https://jaeger.dev.storj.io/trace/afd242eeed2927 FAIL (2025-11-13 14:44:34 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.224850623s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X.X.....................................] duration: 6.127902545s trace: https://jaeger.dev.storj.io/trace/2a65ce5005cafd1d FAIL (2025-11-13 14:44:33 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.127937322s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.204091813s trace: https://jaeger.dev.storj.io/trace/d6ec5f1d62f8624 FAIL (2025-11-13 14:44:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:32.932592910 (102h12m31.484063371s) [upload=89979]: Starting upload Nov 13 14:16:32.932874286 (102h12m31.484344767s) [upload=89979]: Done starting upload Nov 13 14:16:32.933042266 (102h12m31.484512717s) [upload=89979]: Uploading segments... Nov 13 14:16:32.933200005 (102h12m31.484670467s) [upload=89979]: Got next segment. Inline: true Nov 13 14:16:32.933208482 (102h12m31.484678943s) [upload=89979]: Waiting for error group managing segments... Nov 13 14:16:32.933309854 (102h12m31.484780316s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:32.933317860 (102h12m31.484788322s) [upload=89979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 13 14:16:32.933321948 (102h12m31.484792409s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:17:01.188559179 (102h12m59.74002964s) [upload=89979]: Done uploading segments... Nov 13 14:17:01.188737939 (102h12m59.7402084s) [upload=89979]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.204122762s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.565825917s trace: https://jaeger.dev.storj.io/trace/403ba97a5a1cde7d FAIL (2025-11-13 14:44:27 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:00.282283372 (102h11m58.833753833s) [upload=89970]: Starting upload Nov 13 14:16:00.282383773 (102h11m58.833854234s) [upload=89970]: Done starting upload Nov 13 14:16:00.282466601 (102h11m58.833937052s) [upload=89970]: Uploading segments... Nov 13 14:16:00.282681841 (102h11m58.834152302s) [upload=89970]: Got next segment. Inline: false Nov 13 14:16:00.282731475 (102h11m58.834201936s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:00.282737627 (102h11m58.834208088s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 14:16:02.313071205 (102h12m0.864541666s) [seg_pos={0 0}, upload=89970]: Begin upload segment... Nov 13 14:16:02.313149905 (102h12m0.864620376s) [seg_pos={0 0}, upload=89970]: Done begin upload segment. Nov 13 14:16:02.313271697 (102h12m0.864742158s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.313324648 (102h12m0.864795129s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314320904 (102h12m0.865791355s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314744151 (102h12m0.866214642s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315158950 (102h12m0.866629421s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315599659 (102h12m0.867070121s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315928506 (102h12m0.867398967s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.316477040 (102h12m0.867947501s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317049180 (102h12m0.868519651s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317427380 (102h12m0.868897841s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317934746 (102h12m0.869405207s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318376678 (102h12m0.869847139s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318812577 (102h12m0.870283038s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319367303 (102h12m0.870837765s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319957317 (102h12m0.871427768s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.320489491 (102h12m0.871959952s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:03.991369116 (102h12m2.542839567s) [upload=89970]: Next returned nil segment Nov 13 14:16:03.991386198 (102h12m2.542856649s) [upload=89970]: Waiting for error group managing segments... Nov 13 14:16:04.278532114 (102h12m2.830002575s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.279571643 (102h12m2.831042105s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.289684120 (102h12m2.841154581s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.300556964 (102h12m2.852027435s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.301333473 (102h12m2.852803934s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302408349 (102h12m2.853878821s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302631235 (102h12m2.854101716s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.335556962 (102h12m2.887027413s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.346910641 (102h12m2.898381102s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.371810816 (102h12m2.923281277s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377236428 (102h12m2.928706889s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377262828 (102h12m2.928733289s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383693804 (102h12m2.935164265s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383710837 (102h12m2.935181288s) [seg_pos={0 0}, upload=89970]: Segment reached optimal threshold of 13 pieces. Nov 13 14:16:04.383822670 (102h12m2.935293121s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383922450 (102h12m2.935392911s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ): protocol: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection Nov 13 14:16:04.383929954 (102h12m2.935400415s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384043279 (102h12m2.93551374s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection Nov 13 14:16:04.384050112 (102h12m2.935520563s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384066143 (102h12m2.935536604s) [seg_pos={0 0}, upload=89970]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Nov 13 14:16:04.384159561 (102h12m2.935630013s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:16:04.384171734 (102h12m2.935642195s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:16:04.934815134 (102h12m3.486285595s) [upload=89970]: Done uploading segments... Nov 13 14:16:04.934864327 (102h12m3.486334788s) [upload=89970]: Upload finished. err: <nil> Duration: 6.565925096s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)